Abstract
Image processing and computer vision research have embraced deep learning. This paper offers a deep learning infrared-visible image fusion network using a contrastive learning framework and multi-scale structural similarity (MSSSIM). A novel contrastive learning loss combined with MSSSIM loss is introduced. The MSSSIM loss optimizes the mutual information between source and fused images from various viewpoints and resolutions, whereas contrastive loss reduces the artificially generated noise in the feature. The fusion network has an auto-encoder. The encoder extracts features from the infrared and visible images, and the decoder regenerates the fused image. Based on the similarity between the source and fused images, the loss function directs the network to extract silent targets and background textures from infrared and visible images, respectively. The proposed method outperforms the state-of-the-art in both qualitative and quantitative evaluations.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ma, J., Ma, Y., Li, C.: Infrared and visible image fusion methods and applications: a survey. Inf. Fusion 45, 153–178 (2019)
Bavirisetti, D.P., Dhuli, R.: Two-scale image fusion of visible and infrared images using saliency detection. Infrared Phys. Technol. 76, 52–64 (2016)
Wang, X., Yin, J., Zhang, K., Li, S., Yan, J.: Infrared weak-small targets fusion based on latent low-rank representation and DWT. IEEE Access 7, 112 681–112 692 (2019)
Yang, Y., et al.: Infrared and visible image fusion based on infrared background suppression. Opt. Lasers Eng. 164, 107528 (2023)
Li, H., Wu, X.-J.: DenseFuse: a fusion approach to infrared and visible images. IEEE Trans. Image Process. 28(5), 2614–2623 (2018)
Zhu, Z., Yang, X., Lu, R., Shen, T., Xie, X., Zhang, T.: CLF-Net: contrastive learning for infrared and visible image fusion network. IEEE Trans. Instrum. Meas. 71, 1–15 (2022)
Ram Prabhakar, K., Sai Srikar, V., Venkatesh Babu, R.: DeepFuse: a deep unsupervised approach for exposure fusion with extreme exposure image pairs. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4714–4722 (2017)
Hou, R., et al.: VIF-Net: an unsupervised framework for infrared and visible image fusion. IEEE Trans. Comput. Imaging 6, 640–651 (2020)
Xu, H., Zhang, H., Ma, J.: Classification saliency-based rule for visible and infrared image fusion. IEEE Trans. Comput. Imaging 7, 824–836 (2021)
Kang, M., Park, J.: ContraGAN: contrastive learning for conditional image generation. In: Advances in Neural Information Processing Systems, vol. 33, pp. 21 357–21 369 (2020)
Park, T., Efros, A.A., Zhang, R., Zhu, J.-Y.: Contrastive learning for unpaired image-to-image translation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020, Part IX. LNCS, vol. 12354, pp. 319–345. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58545-7_19
Huang, D.-S., Jo, K.-H., Figueroa-García, J.C.: Intelligent Computing Theories and Application: 13th International Conference, ICIC 2017, Liverpool, UK, August 7–10, 2017, Proceedings, Part II, vol. 10362. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63309-1
Spiegl, B.: Contrastive unpaired translation using focal loss for patch classification. arXiv preprint arXiv:2109.12431 (2021)
Andonian, A., Park, T., Russell, B., Isola, P., Zhu, J.-Y., Zhang, R.: Contrastive feature loss for image prediction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1934–1943 (2021)
Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Zhao, H., Gallo, O., Frosio, I., Kautz, J.: Loss functions for image restoration with neural networks. IEEE Trans. Comput. Imaging 3(1), 47–57 (2016)
Toet, A., et al.: TNO image fusion dataset. Figshare. Data (2014)
Ma, J., Yu, W., Liang, P., Li, C., Jiang, J.: FusionGAN: a generative adversarial network for infrared and visible image fusion. Inf. Fusion 48, 11–26 (2019)
Li, H., Wu, X.-J., Kittler, J.: Infrared and visible image fusion using a deep learning framework. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 2705–2710. IEEE (2018)
Guo, H., Ma, Y., Mei, X., Ma, J.: Infrared and visible image fusion based on total variation and augmented Lagrangian. JOSA A 34(11), 1961–1968 (2017)
Liu, Y., Chen, X., Ward, R.K., Wang, Z.J.: Image fusion with convolutional sparse representation. IEEE Signal Process. Lett. 23(12), 1882–1886 (2016)
Zhao, Z., Xu, S., Zhang, C., Liu, J., Li, P., Zhang, J.: DIDFuse: deep image decomposition for infrared and visible image fusion. arXiv preprint arXiv:2003.09210 (2020)
Bavirisetti, D.P., Dhuli, R.: Fusion of infrared and visible sensor images based on anisotropic diffusion and Karhunen-Loeve transform. IEEE Sens. J. 16(1), 203–209 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Gupta, A.K., Barnwal, M., Mishra, D. (2023). A Contrastive Learning Approach for Infrared-Visible Image Fusion. In: Maji, P., Huang, T., Pal, N.R., Chaudhury, S., De, R.K. (eds) Pattern Recognition and Machine Intelligence. PReMI 2023. Lecture Notes in Computer Science, vol 14301. Springer, Cham. https://doi.org/10.1007/978-3-031-45170-6_21
Download citation
DOI: https://doi.org/10.1007/978-3-031-45170-6_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-45169-0
Online ISBN: 978-3-031-45170-6
eBook Packages: Computer ScienceComputer Science (R0)