Cross-Spectral Image Patch Matching by Learning Features of the Spatially Connected Patches in a Shared Space

  • Dou Quan
  • Shuai Fang
  • Xuefeng Liang
  • Shuang WangEmail author
  • Licheng Jiao
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11362)


Cross-spectral image patch matching is a challenging problem due to the significant difference between two images caused by different imaging mechanisms. We consider cross-spectral image patches can be matched because there exists a shared semantic feature space among them, in which the semantic features from different spectral images will be more independent of the spectral domains. To learn this shared feature space, we propose a progressive comparison of spatially connected feature metric learning with a feature discrimination constrain (SCFDM). The progressive comparison of spatially connected feature network keeps the property of each spectral domain in its corresponding low-level feature space, and interacts the cross-spectral features in a high-level feature space. The feature discrimination constrain enforces the framework to refine the shared semantic space for feature extraction. Extensive experiments shows that SCFDM outperforms the state-of-the-art methods on the cross-spectral dataset in terms of FPR95 and the training convergence. Meanwhile, it also demonstrates a better generalizability on a single spectral dataset.

Supplementary material

484515_1_En_8_MOESM1_ESM.pdf (1.2 mb)
Supplementary material 1 (pdf 1228 KB)


  1. 1.
    Aguilera, C.A., Aguilera, F.J., Sappa, A.D., Aguilera, C., Toledo, R.: Learning cross-spectral similarity measures with deep convolutional neural networks. In: CVPR, pp. 1–9 (2016)Google Scholar
  2. 2.
    Aguilera, C.A., Sappa, A.D., Aguilera, C., Toledo, R.: Cross-spectral local descriptors via quadruplet network. Sensors 17(4), 873 (2017)CrossRefGoogle Scholar
  3. 3.
    Balntas, V., Johns, E., Tang, L., Mikolajczyk, K.: PN-Net: conjoined triple deep network for learning local image descriptors. Preprint arXiv:1601.05030 (2016)
  4. 4.
    Bay, H., Ess, A., Tuytelaars, T., Gool, L.V.: Speeded-up robust features (SURF). Comput. Vis. Image Underst. 110(3), 346–359 (2008)CrossRefGoogle Scholar
  5. 5.
    Belongie, S., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. IEEE TPAMI 24(4), 509–522 (2002)CrossRefGoogle Scholar
  6. 6.
    Brown, M., Hua, G., Winder, S.: Discriminative learning of local image descriptors. IEEE TPAMI 33(1), 43–57 (2011)CrossRefGoogle Scholar
  7. 7.
    Brown, M., Susstrunk, S.: Multi-spectral sift for scene category recognition. In: CVPR, pp. 177–184 (2011)Google Scholar
  8. 8.
    Calonder, M., Lepetit, V., Strecha, C., Fua, P.: BRIEF: binary robust independent elementary features. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 778–792. Springer, Heidelberg (2010). Scholar
  9. 9.
    Chopra, S., Hadsell, R., Lecun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: CVPR, pp. 539–546 (2005)Google Scholar
  10. 10.
    Firmenichy, D., Brown, M., Ssstrunk, S.: Multispectral interest points for RGB-NIR image registration. In: ICIP, pp. 181–184 (2011)Google Scholar
  11. 11.
    Han, X., Leung, T., Jia, Y., Sukthankar, R., Berg, A.C.: MatchNet: unifying feature and metric learning for patch-based matching. In: CVPR, pp. 3279–3286 (2015)Google Scholar
  12. 12.
    Juefeixu, F., Pal, D.K., Savvides, M.: NIR-VIS heterogeneous face recognition via cross-spectral joint dictionary learning and reconstruction. In: CVPR, pp. 141–150 (2015)Google Scholar
  13. 13.
    Kim, J., Lee, J.K., Lee, K.M.: Accurate image super-resolution using very deep convolutional networks. In: CVPR, pp. 1646–1654 (2016)Google Scholar
  14. 14.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. Computer Science (2014)Google Scholar
  15. 15.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: NIPS, pp. 1097–1105 (2012)Google Scholar
  16. 16.
    Kumar, B.G., Carneiro, G., Reid, I.: Learning local image descriptors with deep siamese and triplet convolutional networks by minimising global loss functions. In: CVPR, pp. 5385–5394 (2016)Google Scholar
  17. 17.
    Lin, M., Chen, Q., Yan, S.C.: Network in network. Preprint arXiv:1312.4400 (2013)
  18. 18.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. IJCV 60(2), 91–110 (2004)CrossRefGoogle Scholar
  19. 19.
    Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. IEEE TPAMI 27(10), 1615–1630 (2005)CrossRefGoogle Scholar
  20. 20.
    Nie, F., Huang, H., Cai, X., Ding, C.H.: Efficient and robust feature selection via joint \(\ell 2\), 1-norms minimization. In: Advances in Neural Information Processing Systems, pp. 1813–1821 (2010)Google Scholar
  21. 21.
    Pinggera, P., Breckon, T., Bischof, H.: On cross-spectral stereo matching using dense gradient features. In: CVPR (2012)Google Scholar
  22. 22.
    Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: ICCV, pp. 2564–2571 (2012)Google Scholar
  23. 23.
    Zagoruyko, S., Komodakis, N.: Learning to compare image patches via convolutional neural networks. In: CVPR, pp. 4353–4361 (2015)Google Scholar
  24. 24.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. preprint arXiv:1409.1556 (2014)
  25. 25.
    Simoserra, E., Trulls, E., Ferraz, L., Kokkinos, I., Fua, P., Morenonoguer, F.: Discriminative learning of deep convolutional feature point descriptors. In: ICCV, pp. 118–126 (2015)Google Scholar
  26. 26.
    Tian, Y., Fan, B., Wu, F.: L2-Net: deep learning of discriminative patch descriptor in Euclidean space. In: CVPR (2017)Google Scholar
  27. 27.
    Tola, E., Lepetit, V., Fua, P.: Daisy: an efficient dense descriptor applied to wide-baseline stereo. IEEE TPAMI 32(5), 815–830 (2010)CrossRefGoogle Scholar
  28. 28.
    Wang, S., Quan, D., Liang, X., Ning, M., Guo, Y., Jiao, L.: A deep learning framework for remote sensing image registration. ISPRS J. Photogrammetry Remote Sens. 145, 148 (2018)CrossRefGoogle Scholar
  29. 29.
    Winder, S., Hua, G., Brown, M.: Picking the best daisy. In: CVPR, pp. 178–185 (2009)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Intelligent Perception and Image Understanding of Ministry of Education, School of Artificial IntelligenceXidian UniversityXianChina

Personalised recommendations