基于空间多尺度残差网络的红外与可见光图像融合

大气与环境光学学报 ›› 2023, Vol. 18 ›› Issue (5): 469-478.

基于空间多尺度残差网络的红外与可见光图像融合

张亦孟 1, 林伟国 2*

1 北京机电工程总体设计部, 北京 100005; 2 北京化工大学信息科学与技术学院, 北京 100029

收稿日期:2022-02-17 修回日期:2022-04-08 出版日期:2023-09-28 发布日期:2023-10-11
通讯作者: E-mail: linwg@mail.buct.edu.can E-mail:linwg@mail.buct.edu.can
作者简介:张亦孟 (1990- ), 北京人, 工程师, 主要从事图像处理、运载器系统电气综合与健康管理方面的研究。 E-mail: zhangyimeng1990@vip.sina.com
基金资助:
辽宁省应用基础研究计划项目 (2023JH2, 101300239)

Infrared and visible images fusion with spatial multiscale residual networks

ZHANG Yimen 1, LIN Weiguo 2*

1 Beijing System Design Institute of Electro Mechanic Engineering, Beijing 100005, China; 2 College of Information Science and Technology, Beijing University of Chemical Technology, Beijing 100029, China

Received:2022-02-17 Revised:2022-04-08 Online:2023-09-28 Published:2023-10-11
Contact: Lin weiguo E-mail:linwg@mail.buct.edu.can

摘要/Abstract

摘要： 针对如何充分提取和融合红外与可见光图像典型特征的问题, 提出一种基于空间多尺度残差网络的图像融合算法。首先, 将源图像输入基于空间多尺度残差模块组成的编码器网络, 通过源图像重建任务, 训练编码器自动获取重要特征信息的能力；然后, 引入特征金字塔结构, 设计了特征通道自注意力机制, 编码器输出的基础层和细节层进行融合, 减小尺度噪声, 并由解码器重构出融合图像；最后, 利用公开数据集进行定性和定量实验, 证明了改进算法在突出红外图像目标和保留可见光图像纹理细节两方面的优势, 相比于DDcGAN算法, 新算法的标准差和平均梯度分别提升了12.91%和47.41%。

关键词: 图像融合, 自动编码器, 空间多尺度残差模块, 通道自注意力

Abstract: To fully extract and fuse typical features of infrared and visible images, an image fusion algorithm based on spatial multi-scale residual network is proposed. Firstly, the source image is input into an encoder network composed of spatial multi-scale residual modules, and through the task of image reconstruction, the encoder network is trained to automatically obtain important features. Then, a feature pyramid and a channel self-attention are introduced, the output of basic layer and detail layer by the endoder are fused to reduce scale noise, and the fused image is reconstructed by the decoder. Finally, qualitative and quantitative experiments on public datasets are carried out, and it is demonstrated that the imporved algorithm outperforms the alternatives on highlighting infrared image targets and preserving visible image texture details. Compared with the DDcGAN algorithm, the standard deviation and average gradient of the proposed algorithm have been improved by 12.91% and 47.41%, respectively.

Key words: image fusion, auto-encoder, spatial multi-scale residual module, channel self-attention

中图分类号:

TP391.41

张亦孟, 林伟国 . 基于空间多尺度残差网络的红外与可见光图像融合[J]. 大气与环境光学学报, 2023, 18(5): 469-478.

ZHANG Yimen , LIN Weiguo . Infrared and visible images fusion with spatial multiscale residual networks[J]. Journal of Atmospheric and Environmental Optics, 2023, 18(5): 469-478.

参考文献 20

[1]	Castillo J C, Fernández-Caballero A, Serrano-Cuerda J, et al. Smart environment architecture for robust people detection by
	infrared and visible video fusion [J]. Journal of Ambient Intelligence and Humanized Computing, 2017, 8(2): 223-237.
[2]	Tian Y, Yang W J, Wang J. Image fusion using a multi-level image decomposition and fusion method [J]. Applied Optics, 2021,
60	(24): 7466-7479.
[3]	Ma J Y, Ma Y, Li C. Infrared and visible image fusion methods and applications: A survey [J]. Information Fusion, 2019, 45:
15	3-178.
[4]	Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial networks [J]. Communications of the ACM, 2020, 63
(11)	: 139-144.
[5]	Li H, Wu X J. DenseFuse: A fusion approach to infrared and visible images [J]. IEEE Transactions on Image Processing, 2018,
28	(5): 2614-2623.
[6]	Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convolutional networks [C]. 2017 IEEE Conference on
	Computer Vision and Pattern Recognition (CVPR). July 21-26, 2017, Honolulu, HI, USA. IEEE, 2017: 2261-2269.
[7]	Li H, Wu X J, Durrani T. NestFuse: An infrared and visible image fusion architecture based on nest connection and spatial/
	channel attention models [J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9645-9656.
[8]	Zhou Z W, Rahman Siddiquee M M, Tajbakhsh N, et al. UNet++ : A Nested U-Net Architecture for Medical Image
	Segmentation [M]. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Cham:
	Springer International Publishing, 2018: 3-11.
[9]	An W B, Wang H M. Infrared and visible image fusion with supervised convolutional neural network [J]. Optik, 2020, 219:
16	5120.
[10]	Ma J Y, Yu W, Liang P, et al. FusionGAN: A generative adversarial network for infrared and visible image fusion [J].
	Information Fusion, 2019, 48: 11-26.
[11]	Xu D D, Wang Y C, Xu S Y, et al. Infrared and visible image fusion with a generative adversarial network and a residual
	network [J]. Applied Sciences, 2020, 10(2): 554.
[12]	He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition [C]. 2016 IEEE Conference on Computer
	Vision and Pattern Recognition (CVPR). June 27-30, 2016, Las Vegas, NV, USA. IEEE, 2016: 770-778.
[13]	Ma J Y, Xu H, Jiang J J, et al. DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution
	image fusion [J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995.
[14]	Ronneberger O, Fischer P, Brox T. U-net: Convolutional Networks for Biomedical Image Segmentation [M]. Lecture Notes in
	Computer Science. Cham: Springer International Publishing, 2015: 234-241.
[15]	Lin T Y, Dollár P, Girshick R, et al. Feature pyramid networks for object detection [C]. 2017 IEEE Conference on Computer
	Vision and Pattern Recognition (CVPR). July 21-26, 2017, Honolulu, HI, USA. IEEE, 2017: 936-944.
[16]	Liu P F, Zhao H C, Cao F D. Blind deblurring of noisy and blurry images of multi-scale convolutional neural network [J].
	Infrared and Laser Engineering, 2019, 48(4): 300-308.
	刘鹏飞, 赵怀慈, 曹飞道. 多尺度卷积神经网络的噪声模糊图像盲复原 [J]. 红外与激光工程, 2019, 48(4): 300-308.
[17]	Hu J, Shen L, Sun G. Squeeze-and-excitation networks [C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern
	Recognition. June 18-23, 2018, Salt Lake City, UT, USA. IEEE, 2018: 7132-7141.
[18]	Woo S, Park J, Lee J Y, et al. CBAM: Convolutional Block Attention Module [M]. Computer Vision-ECCV 2018. Cham:
	Springer International Publishing, 2018: 3-19.
[19]	Liu P F, Zhao H C, Li P X. Hyperspectral images reconstruction using adversarial networks from single RGB image [J].
	Infrared and Laser Engineering, 2020, 49(S1): 143-150.
	刘鹏飞, 赵怀慈, 李培玄. 对抗网络实现单幅RGB重建高光谱图像 [J]. 红外与激光工程, 2020, 49(S1): 143-150.
[20]	Xu H, Ma J Y, Le Z L, et al. FusionDN: A unified densely connected network for image fusion [J]. Proceedings of the AAAI
	Conference on Artificial Intelligence, 2020, 34(7): 12484-12491.
[21]	Fu Y, Wu X J. A dual-branch network for infrared and visible image fusion [C]. 2020 25th International Conference on Pattern
	Recognition (ICPR). January 10-15, 2021, Milan, Italy. IEEE, 2021: 10675-10680.
[22]	Zhang X C, Ye P, Xiao G. VIFB: A visible and infrared image fusion benchmark [C]. 2020 IEEE/CVF Conference on
	Computer Vision and Pattern Recognition Workshops (CVPRW). June 14-19, 2020, Seattle, WA, USA. IEEE, 2020: 104-105.
[23]	Tian S W, Lin S Z, Lei H W, et al. Multi-band image synchronous super-resolution and fusion method based on improved
	WGAN-GP [J]. Acta Optica Sinica, 2020, 40(20): 2010001.
	田嵩旺, 蔺素珍, 雷海卫, 等. 基于改进WGAN-GP 的多波段图像同步超分与融合方法 [J]. 光学学报, 2020, 40(20):
20	10001.