Skip to main content
SpringerLink
Log in

Dual-Y network: infrared-visible image patches matching via semi-supervised transfer learning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Infrared-visible image patches matching has many applications, such as target recognition, vision-based navigation, and others. At present, deep learning has achieved excellent performance in visible image patches matching. Due to imaging differences in infrared-visible images and the small number of supervised samples, the existing networks cannot solve them well. We propose a Dual-Y network based on semi-supervised transfer learning to address the challenges. Since the infrared and visible image patches have similarities in the same scene, adversarial domain adaptation uses visible images as the source domain and infrared images as the target domain to solve different imaging principles. Through a large number of unannotated samples training, the two convolutional autoencoders in our network can reconstruct the infrared-visible image patches respectively to improve feature representation ability. In the high-level convolution layers, the cross-domain features are extracted by sharing weights in two domains. The adversarial domain adaptation achieves the domain confusion in the high layers. In the low-level convolutional layers, independent weights in two branches can ensure that the domain-related imaging principles can be preserved. Finally, the annotated data are used for supervised training in infrared-visible image patches matching. With the improvements above, the matching accuracy increase by 10.78%, compared with the fine-tuning based on the pre-training module.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Acharya UR, Fujita H, Oh SL, Hagiwara Y., Tan JH, Adam M, Tan RS (2019) Deep convolutional neural network for the automated diagnosis of congestive heart failure using ecg signals. Appl Intell 49 (1):16–27

    Article  Google Scholar 

  2. Badrinarayanan V, Kendall A, Cipolla R (2017) Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans Pat Anal Mach Intell 39(12):2481–2495

    Article  Google Scholar 

  3. Bay H, Tuytelaars T, Gool LJV (2006) Surf:, speeded up robust features. European Conf Comput Vis 3951:404–417

    Google Scholar 

  4. Cheng KS, Lin HY (2015) Automatic target recognition by infrared and visible image matching. In: Iapr international conference on machine vision applications, pp 312–315

  5. Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M, Lempitsky V (2016) Domain-adversarial training of neural networks. J Mach Learn Res 17(1):189–209

    MathSciNet  MATH  Google Scholar 

  6. Ghifary M, Kleijn WB, Zhang M., Balduzzi D, Li W (2016) Deep reconstruction-classification networks for unsupervised domain adaptation. In: European conference on computer vision. Springer, pp 597–613

  7. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville AC, Bengio Y (2014) Generative adversarial nets. In: Advances in neural information processing systems, vol 27, pp 2672–2680

  8. Han X, Leung T, Jia Y, Sukthankar R, Berg AC (2015) Matchnet: unifying feature and metric learning for patch-based matching. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 3279–3286

  9. Hanif MS (2019) Patch match networks: improved two-channel and siamese networks for image patch matching. Pattern Recogn Lett 120:54–61

    Article  Google Scholar 

  10. Jiang B, Chen C, Jin X (2018) Unsupervised domain adaptation with target reconstruction and label confusion in the common subspace. Neural Comput Applic: 1–14

  11. Li DM, Zhang JL (2013) A improved infrared and visible images matching based on surf. Appl Mech Mater: 1637–1640

  12. Li H, Ding W, Cao X, Liu C (2017) Image registration and fusion of visible and infrared integrated camera for medium-altitude unmanned aerial vehicle remote sensing. Remote Sensing 9(5):441

    Article  Google Scholar 

  13. Li X, Aouf N (2013) Sift and surf feature analysis in visible and infrared imaging for uavs. In: 2012 IEEE 11th international conference on cybernetic intelligent systems (CIS), pp 46–51

  14. Li X, Aouf N (2014) Sift and surf feature analysis in visible and infrared imaging for uavs. In: IEEE international conference on cybernetic intelligent systems

  15. Li-E JI, Yang FB, Wang ZS, Chen L (2012) Visible and infrared image matching algorithm based on edge image and surf features. Infrared Technol 34(11):629–635

    Google Scholar 

  16. Liu H, Cocea M (2017) Granular computing based approach for classification towards reduction of bias in ensemble learning. Granular Comput 2(3):131–139

    Article  Google Scholar 

  17. Liu H, Cocea M (2019) Granular computing-based approach of rule learning for binary classification. Granular Comput 4(2): 275–283

    Article  Google Scholar 

  18. Liu H, Cocea M (2019) Nature-inspired framework of ensemble learning for collaborative classification in granular computing context. Granular Comput 4(4):715–724

    Article  Google Scholar 

  19. Liu H, Cocea M, Ding W (2018) Multi-task learning for intelligent data processing in granular computing context. Granular Comput 3(3):257–273

    Article  Google Scholar 

  20. Long M, Cao Y, Cao Z, Wang J, Jordan MI (2019) Transferable representation learning with deep adaptation networks. IEEE Trans Pattern Anal Mach Intell 41(12):3071–3085

    Article  Google Scholar 

  21. Long M, Zhu H, Wang J, Jordan MI (2016) Unsupervised domain adaptation with residual transfer networks. In: NIPS’16 proceedings of the 30th international conference on neural information processing systems, pp 136–144

  22. Long M, Zhu H, Wang J, Jordan MI (2017) Deep transfer learning with joint adaptation networks. In: ICML’17 proceedings of the 34th international conference on machine learning, vol 70, pp 2208–2217

  23. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60 (2):91–110

    Article  Google Scholar 

  24. van der Maaten L, Hinton GE (2008) Visualizing data using t-sne. J Mach Learn Res 9:2579–2605

    MATH  Google Scholar 

  25. Nie W, Liu A, Su Y (2016) Cross-domain semantic transfer from large-scale social media. Multimedia Sys 22(1):75–85

    Article  Google Scholar 

  26. Rublee E, Rabaud V, Konolige K, Bradski GR (2011) Orb: an efficient alternative to sift or surf. In: 2011 international conference on computer vision, pp 2564–2571

  27. Sima AA, Buckley SJ (2013) Optimizing sift for matching of short wave infrared and visible wavelength images. Remote Sens 5(5):2037–2056

    Article  Google Scholar 

  28. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  29. Tzeng E, Hoffman J, Saenko K, Darrell T (2017) Adversarial discriminative domain adaptation. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 2962–2971

  30. Wang M, Deng W (2018) Deep visual domain adaptation: a survey. Neurocomputing 312:135–153

    Article  Google Scholar 

  31. Wang S, Zhao Z, Yu P, Guang Z (2011) Infrared/visible image matching algorithm based on nsct and daisy. In: 2011 4th international congress on image and signal processing, vol 4, pp 2072–2075

  32. Wu X, Du Z, Guo Y, Fujita H (2019) Hierarchical attention based long short-term memory for chinese lyric generation. Appl Intell 49(1):44–52

    Article  Google Scholar 

  33. Yosinski J, Clune J, Bengio Y, Lipson H (2014) How transferable are features in deep neural networks?. In: Advances in neural information processing systems, pp 3320–3328

  34. Zagoruyko S, Komodakis N (2015) Learning to compare image patches via convolutional neural networks. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 4353–4361

  35. Zhao L, Zhou Y, Lu H, Fujita H (2019) Parallel computing method of deep belief networks and its application to traffic flow prediction. Knowledge Based Systems 163:972–987

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Xi’an Microelectronics Technology Institute, 710065, Xi’an, China

    Yuanhong Mao & Zhanzhuang He

Authors
  1. Yuanhong Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhanzhuang He
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuanhong Mao.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mao, Y., He, Z. Dual-Y network: infrared-visible image patches matching via semi-supervised transfer learning. Appl Intell 51, 2188–2197 (2021). https://doi.org/10.1007/s10489-020-01996-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01996-7

Keywords

Search

Navigation