Skip to main content

Challenges in Image Matching for Cultural Heritage: An Overview and Perspective

  • Conference paper
  • First Online:
Image Analysis and Processing. ICIAP 2022 Workshops (ICIAP 2022)

Abstract

Image matching, as the task of finding correspondences in images, is the upstream component of vision and photogrammetric applications aiming at the reconstruction of 3D scenes, their understanding and comparison. Such applications are of special importance in the context of cultural heritage, as they can support archaeologists to digitally preserve, restore and analyze antiquities, but also to compare their changes over time. The success of deep learning, now firmly established, paired with the evolution of computer hardware, has led to many advances in image processing, including image matching. Despite this progress, image matching still offers challenges, in terms of the matching process itself but also on other practical and technical aspects. This paper gives an overview of the current status of the research in image matching with a particular focus on cultural heritage, presenting both strengths and weaknesses of the most recent approaches by means of visual comparisons on exemplar challenging image pairs. Besides assisting researchers and practitioners in the choice of the most suitable solution for a given task, this analysis also suggests lines of research worth to be investigated by the community in the near future.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://simlocmatch.com/ (currently offline).

  2. 2.

    https://www.cs.ubc.ca/research/image-matching-challenge/current/.

  3. 3.

    https://www.vlfeat.org/.

  4. 4.

    https://drive.google.com/drive/folders/1ws1SvRnym3FPh1J6K4lomTIqEsxR5k49.

References

  1. AliceVision: Meshroom: A 3D reconstruction software (2018). https://github.com/alicevision/meshroom

  2. Apollonio, F., Ballabeni, A., Gaiani, M., Remondino, F.: Evaluation of feature-based methods for automated network orientation. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. XL–5, 47–54 (2014)

    Article  Google Scholar 

  3. Arandjelović, R., Zisserman, A.: Three things everyone should know to improve object retrieval. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2012)

    Google Scholar 

  4. Bellavia, F.: SIFT matching by context exposed. IEEE Trans. Pattern Anal. Mach. Intell. (2022). (early access)

    Google Scholar 

  5. Bellavia, F., Colombo, C.: Rethinking the sGLOH descriptor. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 931–944 (2018)

    Article  Google Scholar 

  6. Bellavia, F., Colombo, C.: Is there anything new to say about SIFT matching? Int. J. Comput. Vis. 128(7), 1847–1866 (2020)

    Article  Google Scholar 

  7. Bellavia, F., Mishkin, D.: HarrisZ\(^+\): Harris corner selection for next-gen image matching pipelines. arXiv ePrint 2109.12925 (2021)

    Google Scholar 

  8. Bellavia, F., Morelli, L., Menna, F., Remondino, F.: Image orientation with a hybrid pipeline robust to rotations and wide-baselines. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. XLVI–2/W1–2022, 73–80 (2022)

    Article  Google Scholar 

  9. Chum, O., Werner, T., Matas, J.: Two-view geometry estimation unaffected by a dominant plane. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2005)

    Google Scholar 

  10. DeTone, D., Malisiewicz, T., Rabinovich, A.: Superpoint: self-supervised interest point detection and description. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    Google Scholar 

  11. Dusmanu, M., et al.: D2-Net: a trainable CNN for joint detection and description of local features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  12. Fischler, M., Bolles, R.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  13. Gruen, A., Remondino, F., Zhang, L.: Photogrammetric reconstruction of the Great Buddha of Bamiyan. Afg. Photogramm. Rec. 19(107), 177–199 (2004)

    Article  Google Scholar 

  14. Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000)

    MATH  Google Scholar 

  15. Hartmann, W., Havlena, M., Schindler, K.: Recent developments in large-scale tie-point matching. ISPRS J. Photogramm. Remote Sens. 115, 47–62 (2016)

    Article  Google Scholar 

  16. Jin, Y., et al.: Image matching across wide baselines: from paper to practice. Int. J. Comput. Vis. 129(2), 517–547 (2021)

    Article  Google Scholar 

  17. Lindenberger, P., Sarlin, P., Larsson, V., Pollefeys, M.: Pixel-perfect structure-from-motion with featuremetric refinement. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2021)

    Google Scholar 

  18. Lowe, D.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  19. Ma, J., Jiang, J., Fan, A., Jiang, J., Yan, J.: Image matching from handcrafted to deep features: a survey. Int. J. Comput. Vis. 129(7), 23–79 (2021)

    Article  MathSciNet  Google Scholar 

  20. Maiwald, F., Lehmann, C., Lazariv, T.: Fully automated pose estimation of historical images in the context of 4d geographic information systems utilizing machine learning methods. ISPRS Int. J. Geoinf. 10(11), 748 (2021)

    Article  Google Scholar 

  21. Mishchuk, A., Mishkin, D., Radenovic, F., Matas, J.: Working hard to know your neighbor’s margins: local descriptor learning loss. In: Proceedings of the Conference on Neural Information Processing Systems (NeurIPS) (2017)

    Google Scholar 

  22. Mishkin, D., Radenovic, F., Matas, J.: Repeatability is not enough: learning affine regions via discriminability. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)

    Google Scholar 

  23. Moulon, P., Monasse, P., Perrot, R., Marlet, R.: OpenMVG: Open multiple view geometry. In: Proceedings of International Workshop on Reproducible Research in Pattern Recognition (2016). https://github.com/openMVG/openMVG

  24. OpenMVS: open Multi-View Stereo reconstruction library (2022). https://github.com/cdcseacave/openMVS

  25. Remondino, F., Menna, F., Morelli, L.: Evaluating hand-crafted and learning-based features for photogrammetric applications. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. XLIII–B2–2021, 549–556 (2021)

    Article  Google Scholar 

  26. Remondino, F., Spera, M., Nocerino, E., Menna, F., Nex, F.: State of the art in high density image matching. Photogramm. Rec. 29(146), 144–166 (2014)

    Article  Google Scholar 

  27. Riba, E., Mishkin, D., Ponsa, D., Rublee, E., Bradski, G.: Kornia: an open source differentiable computer vision library for PyTorch. In: Proceedings of IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (2020)

    Google Scholar 

  28. Sarlin, P.E., DeTone, D., Malisiewicz, T., Rabinovich, A.: SuperGlue: learning feature matching with graph neural networks. In: Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Google Scholar 

  29. Schönberger, J.L., Frahm, J.M.: Structure-from-Motion revisited. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016), https://colmap.github.io/

  30. Schönberger, J., Hardmeier, H., Sattler, T., Pollefeys, M.: Comparative evaluation of hand-crafted and learned local features. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)

    Google Scholar 

  31. Stathopoulou, E., Welponer, M., Remondino, F.: Open-source image-based 3D reconstruction pipeline: review, comparison and evaluation. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. XLII-2/W17, 331–338 (2019)

    Google Scholar 

  32. Sun, J., Shen, Z., Wang, Y., Bao, H., Zhou, X.: LoFTR: detector-free local feature matching with transformers. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2021)

    Google Scholar 

  33. Szeliski, R.: Computer Vision: Algorithms and Applications. Springer, Heidelberg (2022). https://doi.org/10.1007/978-1-84882-935-0

    Book  MATH  Google Scholar 

  34. Tian, Y., Balntas, V., Ng, T., Laguna, A.B., Demiris, Y., Mikolajczyk, K.: D2D: keypoint extraction with describe to detect approach. In: Proceedings of the Asian Conference on Computer Vision (ACCV) (2020)

    Google Scholar 

  35. Tian, Y., Fan, B., Wu, F.: L2-Net: deep learning of discriminative patch descriptor in euclidean space. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6128–6136 (2017)

    Google Scholar 

  36. Tyszkiewicz, M.J., Fua, P., Trulls, E.: DISK: learning local features with policy gradient. In: Proceedings of the Conference on Neural Information Processing Systems (NeurIPS) (2020)

    Google Scholar 

  37. Vincent, M., Coughenour, C., Remondino, F., Gutierrez, M., Bendicho, V.L.M., Fritsch, D.: Rekrei: a public platform for digitally preserving lost heritage. In: Proceedings of the 44th CAA Conference (2016)

    Google Scholar 

  38. Yi, K.M., Trulls, E., Lepetit, V., Fua, P.: LIFT: Learned invariant feature transform. In: Proceedings of the European Conference on Computer Vision (ECCV) (2016)

    Google Scholar 

  39. Yi, K.M., Trulls, E., Ono, Y., Lepetit, V., Salzmann, M., Fua, P.: Learning to find good correspondences. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    Google Scholar 

  40. Zhang, J., et al.: Learning two-view correspondences and geometry using order-aware network. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV). pp. 5844–5853 (2019)

    Google Scholar 

Download references

Acknowledgements

The authors would like to thank the Archaeological and Landscape Park of the Valley of the Temples of Agrigento (Italy), the Cultural Heritage Directorate of the Autonomous Province of Trento (Italy) and the Superintendence of the Imperia and Savona provinces (Italy) for providing some of the images used in this work.F. Bellavia is funded by the Italian Ministry of Education and Research (MIUR) under the program PON Ricerca e Innovazione 2014–2020, cofunded by the European Social Fund (ESF), CUP B74I18000220006, id. proposta AIM 1875400, linea di attività 2, Area Cultural Heritage.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to F. Bellavia .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bellavia, F., Colombo, C., Morelli, L., Remondino, F. (2022). Challenges in Image Matching for Cultural Heritage: An Overview and Perspective. In: Mazzeo, P.L., Frontoni, E., Sclaroff, S., Distante, C. (eds) Image Analysis and Processing. ICIAP 2022 Workshops. ICIAP 2022. Lecture Notes in Computer Science, vol 13373. Springer, Cham. https://doi.org/10.1007/978-3-031-13321-3_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-13321-3_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-13320-6

  • Online ISBN: 978-3-031-13321-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics