Skip to main content
SpringerLink
Log in

LTM: efficient learning with triangular topology constraint for feature matching with heavy outliers

  • ORIGINAL PAPER
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

Image feature matching, which aims to establish correspondence between two images, is an important task in computer vision. Among image feature matching, the removal of mismatches is crucial to ensure the correctness of the matches. In recent years, machine learning has become a new perspective for mismatch removal. However, existing learning-based methods require a large amount of image data for training, which shows a lack of generalizability and is hard to deal with cases with high mismatch ratio. In this paper, we induce the triangular topology constraint into machine learning, where topology constraints around the matching points are summarized; combining with the idea of sampling, we achieve the task of removing mismatches. Topology constraints are studied in spite of the image input; our LTM (learning topology for matching) just needs fewer than 20 parameters as input, so that only ten training image pairs from four image sets involving about 3,000 matches are employed to train; it still achieves promising results on various datasets with different machine learning approaches. The experimental results of this study also demonstrate the superior performance of our LTM over existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

All datasets used in the work are publicly available and can be accessed as described in each of the referenced presentation papers.

References

  1. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004). https://doi.org/10.1023/B:VISI.0000029664.99615.94

    Article  Google Scholar 

  2. Bay, H., Tuytelaars, T., Gool, L.V.: SURF: Speeded Up Robust Features. Springer-Verlag, Cham (2006)

    Google Scholar 

  3. C. Minchael, L. Vincent, S. Christoph, V. F. Pascal, BRIEF: binary robust independent elementary features. In: Proceedings of the 11th European Conference on Computer vision: Part IV Sept 2010 Pp. 778–792. https://doi.org/10.1007/978-3-642-15561-1_56.

  4. Yi, K.M., Trulls, E., Lepetit, V., et al.: LIFT: learned invariant feature transform. Eur. Conf. Comput. Vis. (2016). https://doi.org/10.1007/978-3-319-46466-4_28

    Article  Google Scholar 

  5. D. Detone, Malisiewicz T , Rabinovich A. SuperPoint: self-supervised interest point detection and description[C]// In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). https://doi.org/10.1109/CVPRW.2018.00060.

  6. Richard, H., Andrew, Z.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000)

    MATH  Google Scholar 

  7. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981). https://doi.org/10.1145/358669.358692

    Article  MathSciNet  Google Scholar 

  8. Chum O, Matas J. Matching with PROSAC-progressive sample consensus. In: Proceeding of IEEE Computer Society Conference on Computer Society, 2005: 220–226. https://doi.org/10.1109/CVPR.2005.221

  9. Ni Kai, Jin Hailing, Dellaert F. GroupSAC: efficient consensus in the presence of groupings. In: Proceeding of the 12th IEEE International Conference on Computer Vision. 2009:2193–2200. https://doi.org/10.1109/ICCV.2009.5459241

  10. Chun O, Matas J. Randomized RANSAC with Td, d test. In: Proceeding of the 13th British Machine Vision Conference. Berlin: Springer, 2002: 448–457

  11. Matas J, Chun O. Randomized RANSAC with sequential probability ratio test [C]. In: Proc of the 10th IEEE International Conference on Computer Vision (ICCV): IEEE Press, 2005: 1727–1732. https://doi.org/10.1109/ICCV.2005.198

  12. M. Rahman, X. Li and X. Yin. DL-RANSAC: An improved RANSAC with modified sampling strategy based on the likelihood [C]. In: 2019 IEEE 4th International Conference on Image, Vision and Computing (ICIVC). 2019: 463–468. https://doi.org/10.1109/ICIVC47709.2019.8981025

  13. Shi C, Wang Y, Li H, Feature point matching using sequential evaluation on sample consensus. In: International Conference on Security. IEEE, 2017: 302–306. https://doi.org/10.1109/SPAC.2017.8304294

  14. Aguilar, W., Fraud, Y., Escolano, F., et al.: A robust Graph Transformation Matching for non-rigid registration. Image Vis. Comput. 27(7), 897–910 (2009)

    Article  Google Scholar 

  15. Bian, J., Lin, W., Matsushita, Y., Yeung, S., Nguyen, T., Cheng, M.: GMS: grid-based motion statistics for fast, ultra-robust feature correspondence. IEEE Conf. Comput. Vis. Pattern Recogn. (CVPR). 2017, 2828–2837 (2017). https://doi.org/10.1109/CVPR.2017.302

    Article  Google Scholar 

  16. Ma, J., Zhao, J., Jiang, J., et al.: Locality preserving matching. Int. J. Comput. Vision 127(5), 512–531 (2019). https://doi.org/10.1007/s11263-018-1117-z

    Article  MathSciNet  MATH  Google Scholar 

  17. Jiang, X., Xia, Y., Zhang, X.-P., Ma, J.: Robust image matching via local graph structure consensus. Pattern Recogn. 126, 108588 (2022). https://doi.org/10.1016/j.patcog.2022.108588

    Article  Google Scholar 

  18. Zhao, X., He, Z., et al.: Improved keypoint descriptors based on Delaunay triangulation for image matching. Optik-Int. J. Light Electron. 125(13), 3121–3123 (2014). https://doi.org/10.1016/j.ijleo.2013.12.022

    Article  Google Scholar 

  19. Luo Y, Li R, Zhang J, et al. Research on Correction Method of Local Feature Descriptor Mismatch. In: 2019 IEEE 4th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC). https://doi.org/10.1109/IAEAC47372.2019.8997949

  20. Zhao M, Chen H, Song T, et al. Research on image matching based on improved RANSAC-SIFT algorithm. International Conference on Optical Communications and Networks, pp 1–3 (2017). https://doi.org/10.1109/ICOCN.2017.8121270

  21. Zhu, W., Sun, W., Wang, Y., Liu, S., Xu, K.: An improved RANSAC algorithm based on similar structure constraints. Int. Conf. Robot Intell. Syst. (ICRIS) 2016, 94–98 (2016). https://doi.org/10.1109/ICRIS.2016.19

    Article  Google Scholar 

  22. X. Lan, B. Guo, Z. Huang and S. Zhang, "An improved UAV aerial image mosaic algorithm based on GMS-RANSAC". In: 2020 IEEE 5th International Conference on Signal and Image Processing (ICSIP), 2020, pp. 148–152. https://doi.org/10.1109/ICSIP49896.2020.9339283

  23. He, Z., Shen, C., Wang, Q., Zhao, X., Jiang, H.: Mismatching removal for feature-point matching based on triangular topology probability sampling consensus. Remote Sens. 14, 706 (2022). https://doi.org/10.3390/rs14030706

    Article  Google Scholar 

  24. Mikolajczyk, K., Tuytelaars, T., Schmid, C., et al.: A comparison of affine region detectors. Int. J. Comput. Vision 65(1), 43–72 (2005). https://doi.org/10.1007/s11263-005-3848-x

    Article  Google Scholar 

  25. Balntas, V., Lenc, K., Vedaldi, A., Mikolajczyk, K.: HPatches: a benchmark and evaluation of handcrafted and learned local descriptors. IEEE Conf. Comput. Vision Pattern Recog. (CVPR) 2017, 3852–3861 (2017). https://doi.org/10.1109/ICSIP49896.2020.9339283

    Article  Google Scholar 

  26. K. Cordes, B. Rosenhahn, and J. Ostermann. High-resolution feature evaluation benchmark. In: Proc. CAIP, pp. 327– 334, 2013

  27. Jiang, X., Ma, J., Jiang, J., Guo, X.: Robust feature matching using spatial clustering with heavy outliers. IEEE Trans. Image Process. 29, 736–746 (2019). https://doi.org/10.1109/TIP.2019.2934572

    Article  MathSciNet  MATH  Google Scholar 

  28. Shao, F., Liu, Z., An, J.: Feature matching based on minimum relative motion entropy for image registration. IEEE Trans. Geosci. Remote Sens. 60, 1–12 (2022). https://doi.org/10.1109/TGRS.2021.3068185

    Article  Google Scholar 

  29. L. Cavalli, V. Larsson, M. R. Oswald, T. Sattler and M. Pollefeys, "Handcrafted outlier detection revisited", In: Computer Vision–ECCV 2020: 16th European Conference, pp. 770–787, 2020

  30. Yi KM, Trulls E, Ono Y, Lepetit V, Salzmann M, Fua P. Learning to find good correspondences. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018 (pp. 2666-2674) https://doi.org/10.1109/CVPR.2018.00282

  31. Sarlin P.-E., DeTone D., Malisiewicz T. and Rabinovich A., "SuperGlue: learning feature matching with graph neural networks". In: Proceeding of IEEE/CVF Conference of Computer Vision and Pattern Recognition (CVPR), pp. 4938–4947, Jun. 2020. https://doi.org/10.1109/CVPR42600.2020.00499

  32. J. Zhang et al., "OANet: Learning Two-View Correspondences and Geometry Using Order-Aware Network". In: IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 6, pp. 3110–3122, 1 June 2022. https://doi.org/10.1109/TPAMI.2020.3048013

  33. Xia, Y., Ma, J.: Locality-guided global-preserving optimization for robust feature matching. IEEE Trans. Image Process. 31, 5093–5108 (2022). https://doi.org/10.1109/TIP.2022.3192993

    Article  Google Scholar 

  34. Xia, Y., Jiang, J., Yifan, Lu., Liu, W., Ma, J.: Robust feature matching via progressive smoothness consensus. ISPRS J. Photogr. Remote. Sens. 196, 502–513 (2023)

    Article  Google Scholar 

  35. Liu, X., Xiao, G., Chen, R., Ma, J.: PGFNet: preference-guided filtering network for two-view correspondence learning. IEEE Trans. Image Process. 32, 1367–1378 (2023). https://doi.org/10.1109/TIP.2023.3242598

    Article  Google Scholar 

  36. Ma, J., Jiang, X., Jiang, J., Zhao, J., Guo, X.: LMR: learning a two-class classifier for mismatch removal. IEEE Trans. Image Process. 28(8), 4045–4059 (2019). https://doi.org/10.1109/TIP.2019.2906490

    Article  MathSciNet  MATH  Google Scholar 

  37. N. Mayer et al., "A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation". In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 4040–4048. https://doi.org/10.1109/CVPR.2016.438

  38. Liu, Z., An, J., Jing, Y.: A simple and robust feature point matching algorithm based on restricted spatial order constraints for aerial image registration. IEEE Trans. Geosci. Remote Sens. 50(2), 514–527 (2012). https://doi.org/10.1109/TGRS.2011.2160645

    Article  Google Scholar 

  39. Zhang, K., Li, X., Zhang, J.: A robust point-matching algorithm for remote sensing image registration. IEEE Geosci. Remote Sens. Lett. 11(2), 469–473 (2014). https://doi.org/10.1109/LGRS.2013.2267771

    Article  MathSciNet  Google Scholar 

  40. Wu, Y., Ma, W., Gong, M., Su, L., Jiao, L.: A novel point-matching algorithm based on fast sample consensus for image registration. IEEE Geosci. Remote Sens. Lett. 12(1), 43–47 (2015). https://doi.org/10.1109/LGRS.2014.2325970

    Article  Google Scholar 

  41. Li, B., Ye, H.: RSCJ: robust sample consensus judging algorithm for remote sensing image registration. IEEE Geosci. Remote Sens. Lett. 9(4), 574–578 (2012). https://doi.org/10.1109/LGRS.2011.2175434

    Article  Google Scholar 

  42. H. Zhang et al., "Remote sensing image registration based on local affine constraint with circle descriptor." In: IEEE Geoscience and Remote Sensing Letters, vol. 19, pp. 1–5, 2022, Art no. 8002205, https://doi.org/10.1109/LGRS.2020.3027096

  43. Feng, R., Shen, H., Bai, J., Li, X.: Advances and opportunities in remote sensing image geometric registration: a systematic review of state-of-the-art approaches and future research directions. IEEE Geosci Remote Sens Magaz 9(4), 120–142 (2021). https://doi.org/10.1109/MGRS.2021.3081763

    Article  Google Scholar 

Download references

Funding

This work was supported by the National Natural Science Foundation of China (52275547 and 52275514) and the Zhejiang Provincial Natural Science Foundation of China (LY21E050021).

Author information

Authors and Affiliations

  1. School of Mechanical Engineering, the State Key Lab of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou, 310058, People’s Republic of China

    Chentao Shen, Zaixing He & Xinyue Zhao

  2. Zhejiang Feihang Intelligent Technology Co., LTD, Huzhou, 313200, People’s Republic of China

    Wenfeng Cui & Huarong Shen

Authors
  1. Chentao Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zaixing He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinyue Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenfeng Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Huarong Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Zaixing He or Xinyue Zhao.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shen, C., He, Z., Zhao, X. et al. LTM: efficient learning with triangular topology constraint for feature matching with heavy outliers. Machine Vision and Applications 34, 130 (2023). https://doi.org/10.1007/s00138-023-01482-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-023-01482-3

Keywords

Search

Navigation