Skip to main content
SpringerLink
Log in

A Comparative Study of Graph Matching Algorithms in Computer Vision

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13683))

Included in the following conference series:

Abstract

The graph matching optimization problem is an essential component for many tasks in computer vision, such as bringing two deformable objects in correspondence. Naturally, a wide range of applicable algorithms have been proposed in the last decades. Since a common standard benchmark has not been developed, their performance claims are often hard to verify as evaluation on differing problem instances and criteria make the results incomparable. To address these shortcomings, we present a comparative study of graph matching algorithms. We create a uniform benchmark where we collect and categorize a large set of existing and publicly available computer vision graph matching problems in a common format. At the same time we collect and categorize the most popular open-source implementations of graph matching algorithms. Their performance is evaluated in a way that is in line with the best practices for comparing optimization algorithms. The study is designed to be reproducible and extensible to serve as a valuable resource in the future.

Our study provides three notable insights: (i) popular problem instances are exactly solvable in substantially less than 1 s, and, therefore, are insufficient for future empirical evaluations; (ii) the most popular baseline methods are highly inferior to the best available methods; (iii) despite the NP-hardness of the problem, instances coming from vision applications are often solvable in a few seconds even for graphs with more than 500 vertices.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For sets A and B the notation \(x\in A^B\) denotes a vector x whose coordinates take on values from the set A and are indexed by elements of B, i.e., each element of B corresponds to a value from A.

  2. 2.

    The web site for the benchmark is available at https://vislearn.github.io/gmbench/.

  3. 3.

    Strictly speaking, the term doubly-stochastic corresponds to the case when equality constraints are considered in (1). In [65] the inequality variant is called doubly semi-stochastic but we use doubly-stochastic in both cases

  4. 4.

    Non-negative in original maximization formulations.

  5. 5.

    Reparametrized costs are also known as reduced costs, e.g., in the simplex tableau.

  6. 6.

    Most of the considered graphical models are dense in terms of [61].

References

  1. Adams, W.P., Johnson, T.A.: Improved linear programming-based lower bounds for the quadratic assignment problem. Discrete Math. Theor. Comput. Sci. 16, 43–77 (1994)

    Google Scholar 

  2. Ahuja, R.K., Magnanti, T.L., Orlin, J.B.: Network Flows: Theory. Prentice Hall, Algorithms and Applications (1993)

    MATH  Google Scholar 

  3. Alhaija, H.A., Sellent, A., Kondermann, D., Rother, C.: GraphFlow - 6D large displacement scene flow via graph matching. In: Proceedings of the DAGM German Conference on Pattern Recognition (2015)

    Google Scholar 

  4. Alhaija, H.A., Sellent, A., Kondermann, D., Rother, C.: Graph Matching Problems for GraphFlow - 6D Large Displacement Scene Flow Problem Instances (2018), https://research-explorer.app.ist.ac.at/record/5573

  5. Alhaija, H.A., Sellent, A., Kondermann, D., Rother, C.: Project GraphFlow - 6D Large Displacement Scene Flow Images (2018). https://hci.iwr.uni-heidelberg.de/vislearn/research/image-matching/graphflow/

  6. Beiranvand, V., Hare, W., Lucet, Y.: Best practices for comparing optimization algorithms. Optimization and Engineering (2017)

    Google Scholar 

  7. Bernard, F., Thunberg, J., Gemmar, P., Hertel, F., Husch, A., Goncalves, J.: A solution for multi-alignment by transformation synchronisation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)

    Google Scholar 

  8. Bertsekas, D.P.: A distributed algorithm for the assignment problem. Lab. for Information and Decision Systems Working Paper, MIT (1979)

    Google Scholar 

  9. Bertsekas, D.P.: Nonlinear programming, second edition. Athena scientific (1999)

    Google Scholar 

  10. Burkard, R., Dell’Amico, M., Martello, S.: Assignment Problems. SIAM (2009)

    Google Scholar 

  11. Burkard, R., Karisch, S., Rendl, F.: QAPLIB - a quadratic assignment problem library. J. Global Optim. 10, 391–403 (1997)

    Google Scholar 

  12. Drezner, Z.: The quadratic assignment problem. In: Laporte, G., Nickel, S., da Gama, F.S. (eds.) Location Science, pp. 345–363. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-13111-5_13

    Chapter  Google Scholar 

  13. Caetano, T.: Data for Learning Graph Matching (2011). https://www.tiberiocaetano.com/data/

  14. Caetano, T.S., McAuley, J.J., Cheng, L., Le, Q.V., Smola, A.J.: Learning Graph Matching. IEEE Trans. Pattern Anal. Mach. Intell. 31, 1048–1058 (2009)

    Google Scholar 

  15. Cela, E.: The Quadratic Assignment Problem: Theory and Algorithms, vol. 1. Springer Science & Business Media (2013)

    Google Scholar 

  16. Cho, M., Jungmin, L., Kyoung, M.L.: Reweighted Random Walks for Graph Matching: Project Page (2010). https://cv.snu.ac.kr/research/~RRWM/

  17. Cho, M., Lee, J., Lee, K.M.: Reweighted random walks for graph matching. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6315, pp. 492–505. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15555-0_36

    Chapter  Google Scholar 

  18. Cho, M., Sun, J., Duchenne, O., Ponce, J.: Finding matches in a haystack: a max-pooling strategy for graph matching in the presence of outliers. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014)

    Google Scholar 

  19. Cho, M., Sun, J., Duchenne, O., Ponce, J.: Finding matches in a haystack source code (2014). https://www.di.ens.fr/willow/research/maxpoolingmatching/

  20. Cour, T.: Graph Matching Toolbox in MATLAB (2010). http://www.timotheecour.com/software/graph_matching/graph_matching.html

  21. Cour, T., Srinivasan, P., Shi, J.: Balanced graph matching. In: Advances in Neural Information Processing Systems (2007)

    Google Scholar 

  22. Dolan, E.D., Moré, J.J.: Benchmarking optimization software with performance profiles. Mathematical Programming (2002)

    Google Scholar 

  23. Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The PASCAL visual object classes challenge 2007 results (2007). http://www.pascal-network.org/challenges/VOC/voc2007/workshop/index.html

  24. Foggia, P., Percannella, G., Vento, M.: Graph matching and learning in pattern recognition in the last 10 years. Int. J. Pattern Recogn. Artif. Intelli. 28, 1450001 (2014)

    Google Scholar 

  25. Frank, M., Wolfe, P., et al.: An Algorithm for Quadratic Programming. Naval Research Logistics Quarterly (1956)

    Google Scholar 

  26. Globerson, A., Jaakkola, T.S.: Fixing max-product: convergent message passing algorithms for MAP LP-relaxations. In: Advances in Neural Information Processing Systems (2008)

    Google Scholar 

  27. Gold, S., Rangarajan, A.: A graduated assignment algorithm for graph matching. IEEE Trans. Pattern Anal. Mach. Intell. 28, 1450001 (1996)

    Google Scholar 

  28. Guignard, M., Kim, S.: Lagrangean decomposition: a model yielding stronger lagrangean bounds. Mathematical Programming (1987)

    Google Scholar 

  29. Heimann, T., Meinzer, H.P.: Statistical shape models for 3D medical image segmentation: a review. Med. Image Anal. 13, 543–563 (2009)

    Google Scholar 

  30. Horn, R.A., Johnson, C.R.: Matrix Analysis. Cambridge University Press (2012)

    Google Scholar 

  31. Hutschenreiter, L., Haller, S., Feineis, L., Rother, C., Kainmüller, D., Savchynskyy, B.: Fusion moves for graph matching. In: Proceedings of the IEEE International Conference on Computer Vision (2021)

    Google Scholar 

  32. Hutschenreiter, L., Haller, S., Feineis, L., Rother, C., Kainmüller, D., Savchynskyy, B.: Fusion moves for graph matching website (2021). https://vislearn.github.io/libmpopt/iccv2021/

  33. Jiang, B., Tang, J., Ding, C., Luo, B.: A local sparse model for matching problem. In: Proceedings of the AAAI Conference on Artificial Intelligence (2015)

    Google Scholar 

  34. Kainmueller, D., Jug, F., Rother, C., Myers, G.: Active graph matching for automatic joint segmentation and annotation of C. elegans. In: Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (2014)

    Google Scholar 

  35. Kainmueller, D., Jug, F., Rother, C., Myers, G.: Graph matching problems for annotating C. elegans (2017). https://doi.org/10.15479/AT:ISTA:57

  36. Kappes, J.H., et al.: A comparative study of modern inference techniques for structured discrete energy minimization problems. Int. J. Comput. Vis. 115, 155–184 (2015)

    Google Scholar 

  37. Kappes, J.H., et al.: OpenGM Benchmark (2015). http://hciweb2.iwr.uni-heidelberg.de/opengm/index.php?l0=benchmark

  38. Kolmogorov, V.: Feature Correspondence via Graph Matching Source Code (2015). https://pub.ist.ac.at/~vnk/software.html#GRAPH-MATCHING

  39. Komodakis, N., Paragios, N.: Beyond Loose LP-Relaxations: Optimizing MRFs by Repairing Cycles. In: Proceedings of the European Conference on Computer Vision (2008)

    Google Scholar 

  40. Kosowsky, J., Yuille, A.: The invisible hand algorithm: solving the assignment problem with statistical physics. Neural Networks (1994)

    Google Scholar 

  41. Kuhn, H.W.: The Hungarian Method for the Assignment Problem. Naval Research Logistics Quarterly (1955)

    Google Scholar 

  42. Lawler, E.L.: The quadratic assignment problem. Manage. Sci. 9, 586–599 (1963)

    Google Scholar 

  43. Leordeanu, M., Hebert, M.: A Spectral Technique for Correspondence Problems Using Pairwise Constraints. In: Proceedings of the IEEE International Conference on Computer Vision (2005)

    Google Scholar 

  44. Leordeanu, M.: Efficient methods for graph matching and MAP inference (2013). https://sites.google.com/site/graphmatchingmethods/

  45. Leordeanu, M., Hebert, M.: Cars and Motor Models. https://datasets.d2.mpi-inf.mpg.de/discrete_cv_problems/car_motor_graph_matching.zip

  46. Leordeanu, M., Hebert, M., Sukthankar, R.: An Integer projected fixed point method for graph matching and MAP inference. In: Advances in Neural Information Processing Systems (2009)

    Google Scholar 

  47. Leordeanu, M., Sukthankar, R., Hebert, M.: Unsupervised Learning for Graph Matching. International Journal of Computer Vision (2012)

    Google Scholar 

  48. Loiola, E.M., Maia de Abreu, N.M., Boaventura-Netto, P.O., Hahn, P., Querido, T.: An analytical survey for the quadratic assignment problem. Eur. J. Oper. Res. (2007)

    Google Scholar 

  49. Ma, J., Jiang, X., Fan, A., Jiang, J., Yan, J.: Image matching from handcrafted to deep features: a survey. Int. J. Comput. Vis. 129, 23–79 (2021)

    Google Scholar 

  50. Pardalos, P.M., Rendl, F., Wolkowicz, H.: The Quadratic Assignment Problem - A Survey and Recent Developments. Quadratic Assignment and Related Problems (1993)

    Google Scholar 

  51. Rolínek, M., Swoboda, P., Zietlow, D., Paulus, A., Musil, V., Martius, G.: Deep Graph Matching via Blackbox Differentiation of Combinatorial Solvers. In: Proceedings of the European Conference on Computer Vision (2020)

    Google Scholar 

  52. Rother, C., Kolmogorov, V., Lempitsky, V.S., Szummer, M.: Optimizing binary MRFs via extended roof duality. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2007)

    Google Scholar 

  53. Sahni, S.: Computationally related problems. SIAM J. Comput. 3, 262–279 (1974)

    Google Scholar 

  54. Savchynskyy, B.: Discrete graphical models - an optimization perspective. In: Foundations and Trends in Computer Graphics and Vision (2019)

    Google Scholar 

  55. Sun, H., Zhou, W., Fei, M.: A survey on graph matching in computer vision. In: International Congress on Image and Signal Processing. BioMedical Engineering and Informatics (2020)

    Google Scholar 

  56. Swoboda, P.: LPMP Source Code (2021). https://github.com/LPMP/LPMP

  57. Swoboda, P., Rother, C., Abu Alhaija, H., Kainmuller, D., Savchynskyy, B.: A study of lagrangean decompositions and dual ascent solvers for graph matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)

    Google Scholar 

  58. Torresani, L., Kolmogorov, V., Rother, C.: Hotel and House-sparse Models. https://datasets.d2.mpi-inf.mpg.de/discrete_cv_problems/graph_matching_hotel_house.zip

  59. Torresani, L., Kolmogorov, V., Rother, C.: A dual decomposition approach to feature correspondence. IEEE Trans. Pattern Anal. Mach. Intell. 35, 259–271 (2013)

    Google Scholar 

  60. Tourani, S., Shekhovtsov, A., Rother, C., Savchynskyy, B.: MPLP++: fast, parallel dual block-coordinate ascent for dense graphical models. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11208, pp. 264–281. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01225-0_16

    Chapter  Google Scholar 

  61. Tourani, S., Shekhovtsov, A., Rother, C., Savchynskyy, B.: Taxonomy of dual block-coordinate ascent methods for discrete energy minimization. In: Proceedings of the Conference on Artifical Intelligence and Statistics (2020)

    Google Scholar 

  62. Vogelstein, J.T., et al.: Fast approximate quadratic programming for graph matching. PLOS ONE (2015)

    Google Scholar 

  63. Yan, J., Yin, X.C., Lin, W., Deng, C., Zha, H., Yang, X.: A short survey of recent advances in graph matching. In: Proceedings of the ACM International Conference on Multimedia Retrieval (2016)

    Google Scholar 

  64. Yilmaz, A., Javed, O., Shah, M.: Object tracking: a survey. ACM Comput. 38, 13-es (2006)

    Google Scholar 

  65. Zass, R., Shashua, A.: Probabilistic graph and hypergraph matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2008)

    Google Scholar 

  66. Zhang, Z.: HungarianBP: pairwise matching through max-weight bipartite belief propagation source code (2016). https://github.com/zzhang1987/HungarianBP

  67. Zhang, Z., Shi, Q., McAuley, J., Wei, W., Zhang, Y., van den Hengel, A.: Pairwise matching through max-weight bipartite belief propagation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2016)

    Google Scholar 

  68. Zhou, F.: Implementation of Factorized Graph Matching (2018). https://github.com/zhfe99/fgm

  69. Zhou, F., la Torre, F.D.: Factorized graph matching. IEEE Trans. Pattern Anal. Mach. Intell. 38, 1774–1789 (2016)

    Google Scholar 

Download references

Acknowledgements

This work was supported by the DFG grant SA 2640/2-1 and the Helmholtz Information & Data Science School for Health. We thank the ZIH at TU Dresden for providing high performance computing resources.

Author information

Authors and Affiliations

  1. Heidelberg University, Heidelberg, Germany

    Stefan Haller, Lorenz Feineis, Lisa Hutschenreiter, Carsten Rother & Bogdan Savchynskyy

  2. University of Bonn, Bonn, Germany

    Florian Bernard

  3. MDC Berlin, Berlin, Germany

    Dagmar Kainmüller

  4. MPI-INF Saarbrücken, Saarbrücken, Germany

    Paul Swoboda

Authors
  1. Stefan Haller
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lorenz Feineis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lisa Hutschenreiter
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Florian Bernard
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Carsten Rother
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Dagmar Kainmüller
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Paul Swoboda
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Bogdan Savchynskyy
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefan Haller .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 1063 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Haller, S. et al. (2022). A Comparative Study of Graph Matching Algorithms in Computer Vision. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13683. Springer, Cham. https://doi.org/10.1007/978-3-031-20050-2_37

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20050-2_37

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20049-6

  • Online ISBN: 978-3-031-20050-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation