Skip to main content
SpringerLink
Log in

Deep Graph Matching via Blackbox Differentiation of Combinatorial Solvers

  • Conference paper
  • First Online:
Computer Vision – ECCV 2020 (ECCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12373))

Included in the following conference series:

Abstract

Building on recent progress at the intersection of combinatorial optimization and deep learning, we propose an end-to-end trainable architecture for deep graph matching that contains unmodified combinatorial solvers. Using the presence of heavily optimized combinatorial solvers together with some improvements in architecture design, we advance state-of-the-art on deep graph matching benchmarks for keypoint correspondence. In addition, we highlight the conceptual advantages of incorporating solvers into deep learning architectures, such as the possibility of post-processing with a strong multi-graph matching solver or the indifference to changes in the training setting. Finally, we propose two new challenging experimental setups.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Adams, R.P., Zemel, R.S.: Ranking via sinkhorn propagation (2011)

    Google Scholar 

  2. Abu Alhaija, H., Sellent, A., Kondermann, D., Rother, C.: GraphFlow – 6D large displacement scene flow via graph matching. In: Gall, J., Gehler, P., Leibe, B. (eds.) GCPR 2015. LNCS, vol. 9358, pp. 285–296. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24947-6_23

    Chapter  Google Scholar 

  3. Amos, B., Kolter, J.Z.: OptNet: differentiable optimization as a layer in neural networks. In: International Conference on Machine Learning. ICML 2017, pp. 136–145 (2017)

    Google Scholar 

  4. Baker, S., Scharstein, D., Lewis, J.P., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. Int. J. Comput. Vision 92(1), 1–31 (2011). https://doi.org/10.1007/s11263-010-0390-2

    Article  Google Scholar 

  5. Balcan, M., Dick, T., Sandholm, T., Vitercik, E.: Learning to branch. In: International Conference on Machine Learning. ICML 2018, pp. 353–362 (2018)

    Google Scholar 

  6. Battaglia, P., et al.: Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261 (2018)

  7. Bello, I., Pham, H., Le, Q.V., Norouzi, M., Bengio, S.: Neural combinatorial optimization with reinforcement learning. In: International Conference on Learning Representations, Workshop Track. ICLR 2017 (2017)

    Google Scholar 

  8. Bourdev, L., Malik, J.: Poselets: Body part detectors trained using 3D human pose annotations. In: IEEE International Conference on Computer Vision. ICCV 2009, pp. 1365–1372 (2009)

    Google Scholar 

  9. Burkard, R., Dell’Amico, M., Martello, S.: Assignment Problems. Society for Industrial and Applied Mathematics, Philadelphia (2009)

    Book  Google Scholar 

  10. Burkard, R.E., Karisch, S.E., Rendl, F.: QAPLIB-a quadratic assignment problem library. J. Global Optim. 10(4), 391–403 (1997). https://doi.org/10.1023/A:1008293323270

    Article  MathSciNet  MATH  Google Scholar 

  11. Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2D pose estimation using part affinity fields. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2017 (2017)

    Google Scholar 

  12. Chang, J.R., Chen, Y.S.: Pyramid stereo matching network. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2018, pp. 5410–5418 (2018)

    Google Scholar 

  13. Chen, H.T., Lin, H.H., Liu, T.L.: Multi-object tracking using dynamical graph matching. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, vol. 2, pp. II-II. IEEE (2001)

    Google Scholar 

  14. Cho, M., Alahari, K., Ponce, J.: Learning graphs to match. In: IEEE International Conference on Computer Vision. ICCV 2013 (2013)

    Google Scholar 

  15. Delaunay, B.: Sur la sphere vide. Izv. Akad. Nauk SSSR Otdelenie Matematicheskii i Estestvennyka Nauk 7, 793–800 (1934)

    MATH  Google Scholar 

  16. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2009, pp. 248–255 (2009)

    Google Scholar 

  17. Deudon, M., Cournut, P., Lacoste, A., Adulyasak, Y., Rousseau, L.-M.: Learning heuristics for the TSP by policy gradient. In: van Hoeve, W.-J. (ed.) CPAIOR 2018. LNCS, vol. 10848, pp. 170–181. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93031-2_12

    Chapter  Google Scholar 

  18. Duchenne, O., Joulin, A., Ponce, J.: A graph-matching kernel for object categorization. In: 2011 International Conference on Computer Vision, pp. 1792–1799. IEEE (2011)

    Google Scholar 

  19. Elmsallati, A., Clark, C., Kalita, J.: Global alignment of protein-protein interaction networks: a survey. IEEE/ACM Trans. Comput. Biol. Bioinform. 13(4), 689–705 (2016)

    Article  Google Scholar 

  20. Everingham, M., Van Gool, L., Williams, C., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vision 88(2), 303–338 (2010). https://doi.org/10.1007/s11263-009-0275-4

    Article  Google Scholar 

  21. Ferber, A., Wilder, B., Dilkina, B., Tambe, M.: Mipaal: Mixed integer program as a layer. arXiv preprint arXiv:1907.05912 (2019)

  22. Fey, M., Eric Lenssen, J., Weichert, F., Müller, H.: SplineCNN: fast geometric deep learning with continuous b-spline kernels. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2018, pp. 869–877 (2018)

    Google Scholar 

  23. Fey, M., Lenssen, J.E., Morris, C., Masci, J., Kriege, N.M.: Deep graph matching consensus. In: International Conference on Learning Representations. ICLR 2020 (2020)

    Google Scholar 

  24. Fey, M., Lenssen, J.E., Morris, C., Masci, J., Kriege, N.M.: Deep graph matching consensus. https://github.com/rusty1s/deep-graph-matching-consensus (2020). Commit: be1c4c

  25. Gasse, M., Chételat, D., Ferroni, N., Charlin, L., Lodi, A.: Exact combinatorial optimization with graph convolutional neural networks. In: Advances in Neural Information Processing Systems. NIPS 2019, pp. 15554–15566 (2019)

    Google Scholar 

  26. Grohe, M., Rattan, G., Woeginger, G.J.: Graph similarity and approximate isomorphism. In: 43rd International Symposium on Mathematical Foundations of Computer Science (MFCS 2018). Leibniz International Proceedings in Informatics (LIPIcs), vol. 117, pp. 20:1–20:16 (2018)

    Google Scholar 

  27. Jiang, B., Sun, P., Tang, J., Luo, B.: GLMNet: graph learning-matching networks for feature matching. arXiv preprint arXiv:1911.07681 (2019)

  28. Kainmueller, D., Jug, F., Rother, C., Myers, G.: Active graph matching for automatic joint segmentation and annotation of C. elegans. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8673, pp. 81–88. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10404-1_11

    Chapter  Google Scholar 

  29. Khalil, E., Dai, H., Zhang, Y., Dilkina, B., Song, L.: Learning combinatorial optimization algorithms over graphs. In: Advances in Neural Information Processing Systems. NIPS 2017, pp. 6348–6358 (2017)

    Google Scholar 

  30. Khalil, E.B., Bodic, P.L., Song, L., Nemhauser, G., Dilkina, B.: Learning to branch in mixed integer programming. In: AAAI Conference on Artificial Intelligence. AAAI 2016, pp. 724–731 (2016)

    Google Scholar 

  31. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations. ICLR 2014 (2014)

    Google Scholar 

  32. Kool, W., van Hoof, H., Welling, M.: Attention, learn to solve routing problems! In: International Conference on Learning Representations. ICLR 2019 (2019)

    Google Scholar 

  33. Lawler, E.L.: The quadratic assignment problem. Manag. Sci. 9(4), 586–599 (1963)

    Article  MathSciNet  Google Scholar 

  34. Li, Y., Zemel, R., Brockschmidt, M., Tarlow, D.: Gated graph sequence neural networks. In: International Conference on Learning Representations. ICLR 2016 (2016)

    Google Scholar 

  35. Liu, L., Cheung, W.K., Li, X., Liao, L.: Aligning users across social networks using network embedding. In: International Joint Conference on Artificial Intelligence. IJCAI 2016, pp. 1774–1780 (2016)

    Google Scholar 

  36. Luo, W., Schwing, A.G., Urtasun, R.: Efficient deep learning for stereo matching. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2016, pp. 5695–5703 (2016)

    Google Scholar 

  37. Mandi, J., Demirovic, E., Stuckey, P.J., Guns, T.: Smart predict-and-optimize for hard combinatorial optimization problems. arXiv preprint arXiv:1911.10092 (2019)

  38. Min, J., Lee, J., Ponce, J., Cho, M.: SPair-71k: a large-scale benchmark for semantic correspondance. arXiv preprint arXiv:1908.10543 (2019)

  39. Nam, H., Han, B.: Learning multi-domain convolutional neural networks for visual tracking. arXiv preprint arXiv:1510.07945 (2015)

  40. Niculae, V., Martins, A., Blondel, M., Cardie, C.: SparseMAP: differentiable sparse structured inference. In: International Conference on Machine Learning. ICML 2018, pp. 3799–3808 (2018)

    Google Scholar 

  41. Pachauri, D., Kondor, R., Singh, V.: Solving the multi-way matching problem by permutation synchronization. In: Advances in Neural Information Processing Systems. NIPS 2013, pp. 1860–1868 (2013)

    Google Scholar 

  42. Rolínek, M., Musil, V., Paulus, A., Vlastelica, M., Michaelis, C., Martius, G.: Optimizing ranking-based metrics with blackbox differentiation. In: Conference on Computer Vision and Pattern Recognition. CVPR 2020, pp. 7620–7630 (2020)

    Google Scholar 

  43. Sahillioğlu, Y.: Recent advances in shape correspondence. Vis. Comput. 36(8), 1705–1721 (2019). https://doi.org/10.1007/s00371-019-01760-0

    Article  Google Scholar 

  44. Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G.: The graph neural network model. Trans. Neur. Netw. 20(1), 61–80 (2009)

    Article  Google Scholar 

  45. Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: a unified embedding for face recognition and clustering. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2015, pp. 815–823 (2015)

    Google Scholar 

  46. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  47. Sinkhorn, R., Knopp, P.: Concerning nonnegative matrices and doubly stochastic matrices. Pac. J. Math. 21, 343–348 (1967)

    Article  MathSciNet  Google Scholar 

  48. Storvik, G., Dahl, G.: Lagrangian-based methods for finding MAP solutions for MRF models. IEEE Trans. Image Process. 9(3), 469–479 (2000)

    Article  Google Scholar 

  49. Sun, D., Roth, S., Black, M.J.: A quantitative analysis of current practices in optical flow estimation and the principles behind them. Int. J. Comput. Vis. 106(2), 115–137 (2014). https://doi.org/10.1007/s11263-013-0644-x

    Article  Google Scholar 

  50. Sun, D., Yang, X., Liu, M.Y., Kautz, J.: PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2018, June 2018

    Google Scholar 

  51. Swoboda, P., Kuske, J., Savchynskyy, B.: A dual ascent framework for Lagrangean decomposition of combinatorial problems. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2017, pp. 1596–1606 (2017)

    Google Scholar 

  52. Swoboda, P., Mokarian, A., Theobalt, C., Bernard, F., et al.: A convex relaxation for multi-graph matching. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2019, pp. 11156–11165 (2019)

    Google Scholar 

  53. Swoboda, P., Rother, C., Alhaija, H.A., Kainmüller, D., Savchynskyy, B.: A study of Lagrangean decompositions and dual ascent solvers for graph matching. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2016, pp. 7062–7071 (2016)

    Google Scholar 

  54. Torresani, L., Kolmogorov, V., Rother, C.: A dual decomposition approach to feature correspondence. IEEE Trans. Pattern Anal. Mach. Intell. 35(2), 259–271 (2013)

    Article  Google Scholar 

  55. Ufer, N., Ommer, B.: Deep semantic feature matching. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2017, pp. 6914–6923 (2017)

    Google Scholar 

  56. Vlastelica, M., Paulus, A., Musil, V., Martius, G., Rolínek, M.: Differentiation of blackbox combinatorial solvers. In: International Conference on Learning Representations. ICLR 2020 (2020)

    Google Scholar 

  57. Wang, L., Ouyang, W., Wang, X., Lu, H.: Visual tracking with fully convolutional networks. In: IEEE International Conference on Computer Vision. ICCV 2015, pp. 3119–3127 (2015)

    Google Scholar 

  58. Wang, P.W., Donti, P., Wilder, B., Kolter, Z.: SATNet: bridging deep learning and logical reasoning using a differentiable satisfiability solver. In: International Conference on Machine Learning, pp. 6545–6554 (2019)

    Google Scholar 

  59. Wang, R., Yan, J., Yang, X.: Learning combinatorial embedding networks for deep graph matching. In: IEEE International Conference on Computer Vision. ICCV 2019, pp. 3056–3065 (2019)

    Google Scholar 

  60. Wang, R., Yan, J., Yang, X.: Neural graph matching network: learning Lawler’s quadratic assignment problem with extension to hypergraph and multiple-graph matching. arXiv preprint arXiv:1911.11308 (2019)

  61. Yu, T., Wang, R., Yan, J., Li, B.: Learning deep graph matching with channel-independent embedding and Hungarian attention. In: International Conference on Learning Representations. ICLR 2020 (2020)

    Google Scholar 

  62. Zanfir, A., Sminchisescu, C.: Deep learning of graph matching. In: Conference on Computer Vision and Pattern Recognition. CVPR 2018, pp. 2684–2693 (2018)

    Google Scholar 

  63. Zhang, Y., Hare, J., Prügel-Bennett, A.: Learning representations of sets through optimized permutations. arXiv preprint arXiv:1812.03928 (2018)

  64. Zhang, Z., Lee, W.S.: Deep graphical feature learning for the feature matching problem. In: IEEE International Conference on Computer Vision. ICCV 2019 (2019)

    Google Scholar 

  65. Zhang, Z., Shi, Q., McAuley, J., Wei, W., Zhang, Y., van den Hengel, A.: Pairwise matching through max-weight bipartite belief propagation. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2016 (2016)

    Google Scholar 

  66. Zhou, F., la Torre, F.D.: Factorized graph matching. In: IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2012, pp. 127–134 (2012)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Max Planck Institute for Intelligent Systems, Tübingen, Germany

    Michal Rolínek, Dominik Zietlow, Anselm Paulus & Georg Martius

  2. Max Planck Institute for Informatics, Saarbrücken, Germany

    Paul Swoboda

  3. Università degli Studi di Firenze, Florence, Italy

    Vít Musil

Authors
  1. Michal Rolínek
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul Swoboda
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dominik Zietlow
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anselm Paulus
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Vít Musil
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Georg Martius
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michal Rolínek .

Editor information

Editors and Affiliations

  1. University of Oxford, Oxford, UK

    Andrea Vedaldi

  2. Graz University of Technology, Graz, Austria

    Horst Bischof

  3. University of Freiburg, Freiburg im Breisgau, Germany

    Thomas Brox

  4. University of North Carolina at Chapel Hill, Chapel Hill, NC, USA

    Jan-Michael Frahm

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 332 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rolínek, M., Swoboda, P., Zietlow, D., Paulus, A., Musil, V., Martius, G. (2020). Deep Graph Matching via Blackbox Differentiation of Combinatorial Solvers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12373. Springer, Cham. https://doi.org/10.1007/978-3-030-58604-1_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58604-1_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58603-4

  • Online ISBN: 978-3-030-58604-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation