Skip to main content
Book cover

European Conference on Computer Vision

ECCV 2022: Computer Vision – ECCV 2022 pp 370–388Cite as

Self-supervised Learning of Visual Graph Matching

Part of the Lecture Notes in Computer Science book series (LNCS,volume 13683)

Abstract

Despite the rapid progress made by existing graph matching methods, expensive or even unrealistic node-level correspondence labels are often required. Inspired by recent progress in self-supervised contrastive learning, we propose an end-to-end label-free self-supervised contrastive graph matching framework (SCGM). Unlike in vision tasks like classification and segmentation, where the backbone is often forced to extract object instance-level or pixel-level information, we design an extra objective function at node-level on graph data which also considers both the visual appearance and graph structure by node embedding. Further, we propose two-stage augmentation functions on both raw images and extracted graphs to increase the variance, which has been shown effective in self-supervised learning. We conduct experiments on standard graph matching benchmarks, where our method boosts previous state-of-the-arts under both label-free self-supervised and fine-tune settings. Without the ground truth labels for node matching nor the graph/image-level category information, our proposed framework SCGM outperforms several deep graph matching methods. By proper fine-tuning, SCGM can surpass the state-of-the-art supervised deep graph matching methods. Code is available at https://github.com/Thinklab-SJTU/ThinkMatch-SCGM.

Keywords

  • Self-supervise
  • Graph matching
  • Contrastive learning

C. Liu and S. Zhang—Equal Contribution.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-031-20050-2_22
  • Chapter length: 19 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   89.00
Price excludes VAT (USA)
  • ISBN: 978-3-031-20050-2
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   119.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.

Notes

  1. 1.

    Typically matchable graphs are those falling into the same category, like the images of different cats, which is also the main setting of this paper. While it can be more general for graphs, e.g. there is partial matching between graphs.

  2. 2.

    https://github.com/Thinklab-SJTU/ThinkMatch.

  3. 3.

    Unlike ours, GANN [38] utilizes another direction of self-supervised learning, which does not require the amount of data for pre-training.

References

  1. Bourdev, L., Malik, J.: Poselets: body part detectors trained using 3D human pose annotations. In: 2009 IEEE 12th International Conference on Computer Vision, pp. 1365–1372. IEEE (2009)

    Google Scholar 

  2. Chapelle, O., Weston, J., Bottou, L., Vapnik, V.: Vicinal risk minimization. In: Advances in Neural Information Processing Systems, pp. 416–422 (2001)

    Google Scholar 

  3. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR (2020)

    Google Scholar 

  4. Chen, X., He, K.: Exploring simple siamese representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750–15758 (2021)

    Google Scholar 

  5. Cho, M., Alahari, K., Ponce, J.: Learning graphs to match. In: ICCV, pp. 25–32 (2013)

    Google Scholar 

  6. Cho, M., Lee, J., Lee, K.M.: Reweighted random walks for graph matching. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6315, pp. 492–505. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15555-0_36

    CrossRef  Google Scholar 

  7. Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: ICCV, pp. 1422–1430 (2015)

    Google Scholar 

  8. Dosovitskiy, A., et al.: Flownet: learning optical flow with convolutional networks. In: ICCV, pp. 2758–2766 (2015)

    Google Scholar 

  9. Egozi, A., Keller, Y., Guterman, H.: A probabilistic approach to spectral graph matching. TPAMI 35, 18–27 (2013)

    CrossRef  Google Scholar 

  10. Fey, M., Lenssen, J.E., Morris, C., Masci, J., Kriege, N.M.: Deep graph matching consensus. In: ICLR (2020)

    Google Scholar 

  11. Grill, J.B., et al.: Bootstrap your own latent: a new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 (2020)

  12. Guo, M., Chou, E., Huang, D.A., Song, S., Yeung, S., Fei-Fei, L.: Neural graph matching networks for fewshot 3D action recognition. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 653–669 (2018)

    Google Scholar 

  13. Hadsell, R., Chopra, S., LeCun, Y.: Dimensionality reduction by learning an invariant mapping. In: 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), vol. 2, pp. 1735–1742. IEEE (2006)

    Google Scholar 

  14. Hafidi, H., Ghogho, M., Ciblat, P., Swami, A.: Graphcl: contrastive self-supervised learning of graph representations. arXiv preprint arXiv:2007.08025 (2020)

  15. Hassani, K., Khasahmadi, A.H.: Contrastive multi-view representation learning on graphs. In: International Conference on Machine Learning, pp. 4116–4126. PMLR (2020)

    Google Scholar 

  16. He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738 (2020)

    Google Scholar 

  17. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  18. Yu, J.J., Harley, A.W., Derpanis, K.G.: Back to basics: unsupervised learning of optical flow via brightness constancy and motion smoothness. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9915, pp. 3–10. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-49409-8_1

    CrossRef  Google Scholar 

  19. Jiang, B., Sun, P., Luo, B.: Glmnet: graph learning-matching convolutional networks for feature matching. Pattern Recogn. 121, 108167 (2022)

    CrossRef  Google Scholar 

  20. Kalantidis, Y., Sariyildiz, M.B., Pion, N., Weinzaepfel, P., Larlus, D.: Hard negative mixing for contrastive learning. arXiv preprint arXiv:2010.01028 (2020)

  21. Lai, Z., Lu, E., Xie, W.: Mast: a memory-augmented self-supervised tracker. In: CVPR (2020)

    Google Scholar 

  22. Lawrance, A., Lewis, P.: An exponential moving-average sequence and point process (EMA1). J. Appl. Probab. 14(1), 98–113 (1977)

    CrossRef  MATH  Google Scholar 

  23. Lee, K., Zhu, Y., Sohn, K., Li, C.L., Shin, J., Lee, H.: I-mix: a domain-agnostic strategy for contrastive representation learning. arXiv preprint arXiv:2010.08887 (2020)

  24. Leordeanu, M., Hebert, M., Sukthankar, R.: An integer projected fixed point method for graph matching and map inference. In: NIPS (2009)

    Google Scholar 

  25. Liu, Z.Y., Qiao, H., Xu, L.: An extended path following algorithm for graph-matching problem. TPAMI 34(7), 1451–1456 (2012)

    CrossRef  Google Scholar 

  26. Loiola, E.M., de Abreu, N.M.M., Boaventura-Netto, P.O., Hahn, P., Querido, T.: A survey for the quadratic assignment problem. EJOR 176, 657–90 (2007)

    CrossRef  MathSciNet  MATH  Google Scholar 

  27. Meister, S., Hur, J., Roth, S.: Unflow: unsupervised learning of optical flow with a bidirectional census loss. arXiv preprint arXiv:1711.07837 (2017)

  28. Min, J., Lee, J., Ponce, J., Cho, M.: Spair-71k: a large-scale benchmark for semantic correspondence. arXiv preprint arXiv:1908.10543 (2019)

  29. Nowak, A., Villar, S., Bandeira, A., Bruna, J.: Revised note on learning quadratic assignment with graph neural networks. In: DSW (2018)

    Google Scholar 

  30. Pei, W.Y., Yang, C., Meng, L., Hou, J.B., Tian, S., Yin, X.C.: Scene video text tracking with graph matching. IEEE Access 6, 19419–19426 (2018)

    CrossRef  Google Scholar 

  31. Rolínek, M., Swoboda, P., Zietlow, D., Paulus, A., Musil, V., Martius, G.: Deep graph matching via blackbox differentiation of combinatorial solvers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12373, pp. 407–424. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58604-1_25

    CrossRef  Google Scholar 

  32. Sarlin, P.E., DeTone, D., Malisiewicz, T., Rabinovich, A.: Superglue: learning feature matching with graph neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4938–4947 (2020)

    Google Scholar 

  33. Shim, D., Kim, H.J.: Learning a domain-agnostic visual representation for autonomous driving via contrastive loss. arXiv preprint arXiv:2103.05902 (2021)

  34. Shokoufandeh, A., Keselman, Y., Demirci, M.F., Macrini, D., Dickinson, S.: Many-to-many feature matching in object recognition: a review of three approaches. IET Comput. Vis. 6(6), 500–513 (2012)

    CrossRef  Google Scholar 

  35. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  36. Solé-Ribalta, A., Serratosa, F.: Graduated assignment algorithm for multiple graph matching based on a common labeling. Int. J. Pattern Recognit. Artif. Intell. 27(01), 1350001 (2013)

    Google Scholar 

  37. Wang, R., Yan, J., Yang, X.: Learning combinatorial embedding networks for deep graph matching. In: ICCV, pp. 3056–3065 (2019)

    Google Scholar 

  38. Wang, R., Yan, J., Yang, X.: Graduated assignment for joint multi-graph matching and clustering with application to unsupervised graph matching network learning. In: NeurIPS (2020)

    Google Scholar 

  39. Wang, R., Yan, J., Yang, X.: Neural graph matching network: learning lawler’s quadratic assignment problem with extension to hypergraph and multiple-graph matching. TPAMI (2021)

    Google Scholar 

  40. Wang, S., Wang, R., Yao, Z., Shan, S., Chen, X.: Cross-modal scene graph matching for relationship-aware image-text retrieval. In: The IEEE Winter Conference on Applications of Computer Vision, pp. 1508–1517 (2020)

    Google Scholar 

  41. Wang, T., Ling, H., Lang, C., Feng, S.: Graph matching with adaptive and branching path following. IEEE TPAMI 40(12), 2853–2867 (2017)

    CrossRef  Google Scholar 

  42. Wang, T., Jiang, Z., Yan, J.: Clustering-aware multiple graph matching via decayed pairwise matching composition. In: AAAI (2020)

    Google Scholar 

  43. Wang, T., Isola, P.: Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In: International Conference on Machine Learning, pp. 9929–9939. PMLR (2020)

    Google Scholar 

  44. Wang, X., Zhang, R., Shen, C., Kong, T., Li, L.: Dense contrastive learning for self-supervised visual pre-training. In: ICCV (2021)

    Google Scholar 

  45. Wang, Y., Yang, Y., Yang, Z., Zhao, L., Xu, W.: Occlusion aware unsupervised learning of optical flow. In: CVPR, pp. 4884–4893 (2018)

    Google Scholar 

  46. Wang, Y., Solomon, J.M.: Prnet: self-supervised learning for partial-to-partial registration. arXiv preprint arXiv:1910.12240 (2019)

  47. Wang, Z., et al.: Exploring set similarity for dense self-supervised representation learning. arXiv preprint arXiv:2107.08712 (2021)

  48. Wu, Q., Wan, J., Chan, A.B.: Progressive unsupervised learning for visual object tracking. In: CVPR (2021)

    Google Scholar 

  49. Xie, Z., Lin, Y., Zhang, Z., Cao, Y., Lin, S., Hu, H.: Propagate yourself: exploring pixel-level consistency for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16684–16693 (2021)

    Google Scholar 

  50. Yan, J., Tian, Y., Zha, H., Yang, X., Zhang, Y., Chu, S.: Joint optimization for consistent multiple graph matching. In: ICCV (2013)

    Google Scholar 

  51. Yan, J., Zhang, C., Zha, H., Liu, W., Yang, X., Chu, S.: Discrete hyper-graph matching. In: CVPR (2015)

    Google Scholar 

  52. Yan, J., Yang, S., Hancock, E.: Learning graph matching and related combinatorial optimization problems. In: IJCAI (2020)

    Google Scholar 

  53. Yu, T., Wang, R., Yan, J., Li, B.: Learning deep graph matching with channel-independent embedding and Hungarian attention. In: ICLR (2019)

    Google Scholar 

  54. Zanfir, A., Sminchisescu, C.: Deep learning of graph matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2684–2693 (2018)

    Google Scholar 

  55. Zbontar, J., Jing, L., Misra, I., LeCun, Y., Deny, S.: Barlow twins: self-supervised learning via redundancy reduction. arXiv preprint arXiv:2103.03230 (2021)

  56. Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017)

  57. Zhang, Z., Lee, W.S.: Deep graphical feature learning for the feature matching problem. In: ICCV, pp. 5087–5096 (2019)

    Google Scholar 

  58. Zhao, K., Tu, S., Xu, L.: IA-GM: a deep bidirectional learning method for graph matching. In: AAAI (2021)

    Google Scholar 

  59. Zou, Y., Luo, Z., Huang, J.-B.: DF-Net: unsupervised joint learning of depth and flow using cross-task consistency. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11209, pp. 38–55. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01228-1_3

    CrossRef  Google Scholar 

Download references

Acknowledgements

This work was supported in part by National Key Research and Development Program of China (2020AAA0107600), National Science of Foundation China (61972250, 72061127003), and Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102).

Author information

Authors and Affiliations

  1. MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China

    Chang Liu, Shaofeng Zhang, Xiaokang Yang & Junchi Yan

Authors
  1. Chang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shaofeng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaokang Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Junchi Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junchi Yan .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Liu, C., Zhang, S., Yang, X., Yan, J. (2022). Self-supervised Learning of Visual Graph Matching. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13683. Springer, Cham. https://doi.org/10.1007/978-3-031-20050-2_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20050-2_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20049-6

  • Online ISBN: 978-3-031-20050-2

  • eBook Packages: Computer ScienceComputer Science (R0)