Infrared and visible images fusion with spatial multiscale
residual networks

Abstract

Abstract: To fully extract and fuse typical features of infrared and visible images, an image fusion algorithm based on spatial multi-scale residual network is proposed. Firstly, the source image is input into an encoder network composed of spatial multi-scale residual modules, and through the task of image reconstruction, the encoder network is trained to automatically obtain important features. Then, a feature pyramid and a channel self-attention are introduced, the output of basic layer and detail layer by the endoder are fused to reduce scale noise, and the fused image is reconstructed by the decoder. Finally, qualitative and quantitative experiments on public datasets are carried out, and it is demonstrated that the imporved algorithm outperforms the alternatives on highlighting infrared image targets and preserving visible image texture details. Compared with the DDcGAN algorithm, the standard deviation and average gradient of the proposed algorithm have been improved by 12.91% and 47.41%, respectively.

Key words: image fusion, auto-encoder, spatial multi-scale residual module, channel self-attention

CLC Number:

TP391.41

ZHANG Yimen , LIN Weiguo . Infrared and visible images fusion with spatial multiscale residual networks[J]. Journal of Atmospheric and Environmental Optics, 2023, 18(5): 469-478.

References 20

[1]	Castillo J C, Fernández-Caballero A, Serrano-Cuerda J, et al. Smart environment architecture for robust people detection by
	infrared and visible video fusion [J]. Journal of Ambient Intelligence and Humanized Computing, 2017, 8(2): 223-237.
[2]	Tian Y, Yang W J, Wang J. Image fusion using a multi-level image decomposition and fusion method [J]. Applied Optics, 2021,
60	(24): 7466-7479.
[3]	Ma J Y, Ma Y, Li C. Infrared and visible image fusion methods and applications: A survey [J]. Information Fusion, 2019, 45:
15	3-178.
[4]	Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial networks [J]. Communications of the ACM, 2020, 63
(11)	: 139-144.
[5]	Li H, Wu X J. DenseFuse: A fusion approach to infrared and visible images [J]. IEEE Transactions on Image Processing, 2018,
28	(5): 2614-2623.
[6]	Huang G, Liu Z, Van Der Maaten L, et al. Densely connected convolutional networks [C]. 2017 IEEE Conference on
	Computer Vision and Pattern Recognition (CVPR). July 21-26, 2017, Honolulu, HI, USA. IEEE, 2017: 2261-2269.
[7]	Li H, Wu X J, Durrani T. NestFuse: An infrared and visible image fusion architecture based on nest connection and spatial/
	channel attention models [J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9645-9656.
[8]	Zhou Z W, Rahman Siddiquee M M, Tajbakhsh N, et al. UNet++ : A Nested U-Net Architecture for Medical Image
	Segmentation [M]. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. Cham:
	Springer International Publishing, 2018: 3-11.
[9]	An W B, Wang H M. Infrared and visible image fusion with supervised convolutional neural network [J]. Optik, 2020, 219:
16	5120.
[10]	Ma J Y, Yu W, Liang P, et al. FusionGAN: A generative adversarial network for infrared and visible image fusion [J].
	Information Fusion, 2019, 48: 11-26.
[11]	Xu D D, Wang Y C, Xu S Y, et al. Infrared and visible image fusion with a generative adversarial network and a residual
	network [J]. Applied Sciences, 2020, 10(2): 554.
[12]	He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition [C]. 2016 IEEE Conference on Computer
	Vision and Pattern Recognition (CVPR). June 27-30, 2016, Las Vegas, NV, USA. IEEE, 2016: 770-778.
[13]	Ma J Y, Xu H, Jiang J J, et al. DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution
	image fusion [J]. IEEE Transactions on Image Processing, 2020, 29: 4980-4995.
[14]	Ronneberger O, Fischer P, Brox T. U-net: Convolutional Networks for Biomedical Image Segmentation [M]. Lecture Notes in
	Computer Science. Cham: Springer International Publishing, 2015: 234-241.
[15]	Lin T Y, Dollár P, Girshick R, et al. Feature pyramid networks for object detection [C]. 2017 IEEE Conference on Computer
	Vision and Pattern Recognition (CVPR). July 21-26, 2017, Honolulu, HI, USA. IEEE, 2017: 936-944.
[16]	Liu P F, Zhao H C, Cao F D. Blind deblurring of noisy and blurry images of multi-scale convolutional neural network [J].
	Infrared and Laser Engineering, 2019, 48(4): 300-308.
	刘鹏飞, 赵怀慈, 曹飞道. 多尺度卷积神经网络的噪声模糊图像盲复原 [J]. 红外与激光工程, 2019, 48(4): 300-308.
[17]	Hu J, Shen L, Sun G. Squeeze-and-excitation networks [C]. 2018 IEEE/CVF Conference on Computer Vision and Pattern
	Recognition. June 18-23, 2018, Salt Lake City, UT, USA. IEEE, 2018: 7132-7141.
[18]	Woo S, Park J, Lee J Y, et al. CBAM: Convolutional Block Attention Module [M]. Computer Vision-ECCV 2018. Cham:
	Springer International Publishing, 2018: 3-19.
[19]	Liu P F, Zhao H C, Li P X. Hyperspectral images reconstruction using adversarial networks from single RGB image [J].
	Infrared and Laser Engineering, 2020, 49(S1): 143-150.
	刘鹏飞, 赵怀慈, 李培玄. 对抗网络实现单幅RGB重建高光谱图像 [J]. 红外与激光工程, 2020, 49(S1): 143-150.
[20]	Xu H, Ma J Y, Le Z L, et al. FusionDN: A unified densely connected network for image fusion [J]. Proceedings of the AAAI
	Conference on Artificial Intelligence, 2020, 34(7): 12484-12491.
[21]	Fu Y, Wu X J. A dual-branch network for infrared and visible image fusion [C]. 2020 25th International Conference on Pattern
	Recognition (ICPR). January 10-15, 2021, Milan, Italy. IEEE, 2021: 10675-10680.
[22]	Zhang X C, Ye P, Xiao G. VIFB: A visible and infrared image fusion benchmark [C]. 2020 IEEE/CVF Conference on
	Computer Vision and Pattern Recognition Workshops (CVPRW). June 14-19, 2020, Seattle, WA, USA. IEEE, 2020: 104-105.
[23]	Tian S W, Lin S Z, Lei H W, et al. Multi-band image synchronous super-resolution and fusion method based on improved
	WGAN-GP [J]. Acta Optica Sinica, 2020, 40(20): 2010001.
	田嵩旺, 蔺素珍, 雷海卫, 等. 基于改进WGAN-GP 的多波段图像同步超分与融合方法 [J]. 光学学报, 2020, 40(20):
20	10001.

Infrared and visible images fusion with spatial multiscale residual networks

PDF (PC)

Knowledge

Abstract

Cite this article

share this article

References 20

Related Articles 2

Recommended Articles

Metrics

Comments

[1]	LIANG Yuanan,YI Weining,HUANG Honglian. Detection of Target on Ocean Background Based on Polarization Imagery Fusion [J]. Journal of Atmospheric and Environmental Optics, 2016, 11(1): 60-67.
[2]	ZENG Xian-Fang, XU Guo-Ming, YI Wei-Ning, HUANG Hong-Lian, YIN Cheng-Liang. Polarimetric Image Fast Fusion Method Via Sparse Non-Negative Matrix Factorization [J]. Journal of Atmospheric and Environmental Optics, 2014, 9(3): 229-236.