Abstract
Multi-source fusion is an important research in image target recognition. Different image sources usually can provide complementary knowledge for improving the classification performance. Current methods generally extract features or recognize each source separately before performing fusion, and this cannot well exploit the correlation of different sources. We propose a multi-source image (i.e., visible and infrared images) fusion target recognition method based on mutual learning (MIF-ML). In this paper, an end-to-end visible-infrared image fusion model is constructed. Firstly, two networks are built for the visible and infrared images, respectively, and jointly trained based on mutual learning. The generalization performance of the networks can be efficiently enhanced because the information of different images is transferred between the two networks. Secondly, a weighted decision-level fusion method is developed to combine the classification results of visible and infrared images for achieving as good as possible recognition performance. In the training process, the weight of each image is automatically optimized in the networks. Finally, the performance of the MIF-ML method has been tested by comparing with other related methods, and the experimental results show that the proposed MIF-ML can efficiently improve the classification accuracy.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The datasets used in this article are publicly available and corresponding hyperlinks mentioned in the references.
References
Akram T, Laurent B, Naqvi SR et al (2018) A deep heterogeneous feature fusion approach for automatic land-use classification. Inf Sci 467:199–218
Aziz K, Bouchara F (2018) Multimodal deep learning for robust recognizing maritime imagery in the visible and infrared spectrums. In: International conference image analysis and recognition. Springer, Berlin, pp 235–244
Bhosale YH, Patnaik KS (2022) Application of deep learning techniques in diagnosis of covid-19 (coronavirus): a systematic review. Neural Process Lett 16:1–53
Bhosale YH, Patnaik KS (2022b) Iot deployable lightweight deep learning application for covid-19 detection with lung diseases using raspberrypi. In: 2022 international conference on IoT and blockchain technology (ICIBT), IEEE, pp 1–6
Brown M, Süsstrunk S (2011) Multi-spectral sift for scene category recognition. In: CVPR 2011, IEEE, pp 177–184. https://ivrlwww.epfl.ch/supplementary_material/cvpr11/index.html
Chen Y, Li C, Ghamisi P et al (2017) Deep fusion of remote sensing data for accurate classification. IEEE Geosci Remote Sens Lett 14(8):1253–1257
Chen Z, Li J, Zheng WS (2019) Weakly supervised tracklet person re-identification by deep feature-wise mutual learning https://arxiv.org/abs/arXiv:1910.14333
Deng H, Geng Z, Himed B (2018) Radar target detection using target features and artificial intelligence. In: 2018 international conference on radar (RADAR), IEEE, pp 1–4
Dharini S, Jain S (2021) A novel metaheuristic optimal feature selection framework for object detection with improved detection accuracy based on pulse-coupled neural network. Soft Comput. https://doi.org/10.1007/s00500-021-06126-0
Ding L, Wang Y, Laganière R et al (2021) A robust and fast multispectral pedestrian detection deep network. Knowl-Based Syst 227(106):990
Dong Y, Liu Q, Du B et al (2022) Weighted feature fusion of convolutional neural network and graph attention network for hyperspectral image classification. IEEE Trans Image Process 31:1559–1572. https://doi.org/10.1109/TIP.2022.3144017
Du J, Lu H, Zhang L et al (2021) A spatial-temporal feature-based detection framework for infrared dim small target. IEEE Trans Geosci Remote Sens 60:1–12
Fan J, Chen T, Lu S (2017) Unsupervised feature learning for land-use scene recognition. IEEE Trans Geosci Remote Sens 55(4):2250–2261
Gebremeskel GB (2022) A critical analysis of the multi-focus image fusion using discrete wavelet transform and computer vision. Soft Comput 26(11):5209–5225
He K, Zhang X, Ren S, et al (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
Jia S, Zhan Z, Zhang M et al (2020) Multiple feature-based superpixel-level decision fusion for hyperspectral and lidar data classification. IEEE Trans Geosci Remote Sens 59(2):1437–1452
Kulkarni SC, Rege PP (2020) Pixel level fusion techniques for SAR and optical images: a review. Inf Fusion 59:13–29
Lai X, Qu Y (2019) Adversarial deep mutual learning. In: 2019 IEEE international conference on unmanned systems and artificial intelligence (ICUSAI), IEEE, pp 324–329
Li H, Qi X, Xie W (2020) Fast infrared and visible image fusion with structural decomposition. Knowl-Based Syst 204(106):182
Li X, Zhou F, Tan H (2021) Joint image fusion and denoising via three-layer decomposition and sparse representation. Knowl-Based Syst 224(107):087
Liu Y, Lu Z, Li J et al (2019) Deep image-to-video adaptation and fusion networks for action recognition. IEEE Trans Image Process 29:3168–3182
Liu X, Jiao L, Li L et al (2021) Deep multi-level fusion network for multi-source image pixel-wise classification. Knowl-Based Syst 221(106):921
Liu T, Li J, Wu J et al (2022a) Tracking with mutual attention network. IEEE Trans Multimedia. https://doi.org/10.1109/TMM.2022.3190679
Liu Z, Wang L, Wen Z et al (2022b) Multi-level scattering center and deep feature fusion learning framework for SAR target recognition. IEEE Trans Geosci Remote Sens. https://doi.org/10.1109/TGRS.2022.3174703
Ma J, Ma Y, Li C (2019) Infrared and visible image fusion methods and applications: a survey. Inf Fusion 45:153–178
More SA, Deore PJ (2018) Gait recognition by cross wavelet transform and graph model. IEEE/CAA J Autom Sin 5(3):718–726
Peng X, Li Y, Wei X et al (2018) RGB-NIR image categorization with prior knowledge transfer. EURASIP J Image and Video Process 1:1–11
Qiu X, Li M, Zhang L et al (2020) Deep convolutional feature fusion model for multispectral maritime imagery ship recognition. J Comput Commun 8(11):23
Salamati N, Larlus D, Csurka G (2011) Combining visible and near-infrared cues for image categorisation. In: Proceedings of the 22nd British machine vision conference (BMVC 2011)., CONF
Santos CE, Bhanu B (2018) Dyfusion: dynamic IR/RGB fusion for maritime vessel recognition. In: 2018 25th IEEE international conference on image processing (ICIP), IEEE, pp 1328–1332
Shi Y, Jiang X, Li S (2023) Fusion algorithm of UAV infrared image and visible image registration. Soft Comput 27(2):1061–1073
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556
Uma Maheswari K, Rajesh S (2020) A novel QIM-DCT based fusion approach for classification of remote sensing images via PSO and SVM models. Soft Comput 24(20):15561–15576
Wang Y, Chen Z, Wu Q et al (2020) Deep mutual learning network for gait recognition. Multimedia Tools Appl 79(31):22653–22672
Xiao Y, Wu J, Yuan J (2013) mcentrist: a multi-channel feature generation mechanism for scene categorization. IEEE Trans Image Process 23(2):823–836
Xing L, Shao S, Ma Y et al (2022) Learning to cooperate: Decision fusion method for few-shot remote sensing scene classification. In: IEEE geoscience and remote sensing letters
Xue Q, Zhang W, Zha H (2020) Improving domain-adapted sentiment classification by deep adversarial mutual learning. In: Proceedings of the AAAI conference on artificial intelligence, pp 9362–9369
Zhang MM, Choi J, Daniilidis K et al (2015) Vais: a dataset for recognizing maritime imagery in the visible and infrared spectrums. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 10–16 http://vcipl-okstate.org/pbvs/bench/
Zhang Y, Xiang T, Hospedales TM et al (2018) Deep mutual learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4320–4328
Zhang E, Wang K, Lin G (2019) Classification of marine vessels with multi-feature structure fusion. Appl Sci 9(10):2153
Zhang T, Zhang X, Ke X et al (2022a) Hog-shipclsnet: a novel deep learning network with hog feature fusion for sar ship classification. IEEE Trans Geosci Remote Sens 60:1–22. https://doi.org/10.1109/TGRS.2021.3082759
Zhang Z, Ye S, Zhang Y et al (2022b) Belief combination of classifiers for incomplete data. IEEE/CAA J Autom Sin 9(4):652–667
Funding
This work was supported by the National Natural Science Foundation of China (Grant U20B2067, Grant 61790552 and Grant 61790554).
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Material preparation, data collection, and analysis were performed by [SW], [YY], [ZL], and [QP]. The first draft of the manuscript was written by [Shuyue Wang], and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors have no relevant financial or non-financial interests to disclose.
Ethical approval
Not applicable.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wang, S., Yang, Y., Liu, Z. et al. Target recognition with fusion of visible and infrared images based on mutual learning. Soft Comput 27, 7879–7894 (2023). https://doi.org/10.1007/s00500-023-08010-5
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-023-08010-5