Infrared and visible image fusion based on NSCT and stacked sparse autoencoders
To integrate the infrared object into the fused image effectively, a novel infrared (IR) and visible (VI) image fusion method by using nonsubsampled contourlet transform (NSCT) and stacked sparse autoencoders (SSAE) is proposed. Firstly, the IR and VI images are decomposed into low-frequency subbands and high-frequency subbands by using NSCT. Secondly, SSAE is performed on the low frequency subband of IR image to calculate the object reliabilities (OR) of the low frequency subband coefficients. Subsequently, an adaptive multi-strategy fusion rule based on OR is designed for the fusion of low frequency subbands and a choose-max fusion rule with the absolute values of high frequency subband coefficients are employed for the fusion of high frequency subbands. Experimental results show the proposed method is superior to the conventional methods in highlighting the infrared objects as well as keeping the background information in VI image.
KeywordsImage fusion Stacked sparse autoencoders Nonsubsampled contourlet transform Infrared images
This work was supported by the National Natural Science Foundation of P. R. China under grant no.61772237, the Provincial research grant no. BK20151358, BK20151202, the Suzhou science and technology project under Grant SYG201702, the Fundamental Research Funds for the Central Universities JUSRP51618B and the Equipment Development and Ministry of Education union fund 6141A02033312.
- 5.Chen L, Zhang H, Xiao J, Nie L, Shao J, Liu W, Chua T-S (2017) SCA-CNN: Spatial and Channel-Wise attention in convolutional networks for image captioning. IEEE International Conference on Computer Vision 2017:6298–6306Google Scholar
- 9.Eckhorn R, Reitboeck HJ, Arndt M, Dicke P (1989) A neural network for feature linking via synchronous activity: Results from cat visual cortex and from simulations. Models of Brain Function. Cambridge University Press, pp 255–272Google Scholar
- 10.Fu Z, Dai X, Li Y, Wu H, Wang X (2014) An Improved visible and infrared image fusion based on local energy and fuzzy logic. In: Signal processing (ICSP), pp 861–865Google Scholar
- 14.Geng X, Zhang H, Bian J, Chua T-S (2015) Learning image and user features for recommendation in social networks. In: IEEE International conference on computer vision, pp 4274-4282Google Scholar
- 15.Geng P, Sun X, Liu J (2017) Adopting quaternion wavelet transform to fuse Multi-Modal medical images. Multimed Tools Appl 37(2):230–239Google Scholar
- 21.Lu B, Miao C (2010) Structure tensor based image fusion, Proceedings of the International Symposium on Electronic CommercGoogle Scholar
- 25.Ranzato M, Poultney CS, Chopra S, Lecun Y (2006) Efficient learning of sparse representations with an Energy-Based model. Adv Neural Inf Process Syst 19:1137–1144Google Scholar
- 28.Wang M, Chen Y, Wang X (2014) Recognition of Handwritten Characters in Chinese Legal Amounts by Stacked Autoencoders, 2014 22nd International Conference on Pattern Recognition, pp 3002–3007Google Scholar
- 43.Zhang H, Kyaw Z, Chang S-F, Chua T-S (2017) Visual translation embedding network for visual relation detection. In: IEEE International conference on computer vision and pattern recognition, pp 3107–3115Google Scholar
- 44.Zhang H, Kyaw Z, Yu J, Chang SF (2017) PPR-FCN: Weakly supervised visual relation detection via parallel pairwise r-FCN. IEEE International Conference on Computer Vision 2017:4243–4251Google Scholar
- 47.Zhu L, Shen J, Liu X, Xie L, Nie L (2016) Learning compact visual representation with canonical views for robust mobile landmark search. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI-16), pp 3959-3965Google Scholar