Skip to main content
SpringerLink
Log in

Fast and Differentiable Message Passing on Pairwise Markov Random Fields

  • Conference paper
  • First Online:
Computer Vision – ACCV 2020 (ACCV 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12624))

Included in the following conference series:

Abstract

Despite the availability of many Markov Random Field (MRF) optimization algorithms, their widespread usage is currently limited due to imperfect MRF modelling arising from hand-crafted model parameters and the selection of inferior inference algorithm. In addition to differentiability, the two main aspects that enable learning these model parameters are the forward and backward propagation time of the MRF optimization algorithm and its inference capabilities. In this work, we introduce two fast and differentiable message passing algorithms, namely, Iterative Semi-Global Matching Revised (ISGMR) and Parallel Tree-Reweighted Message Passing (TRWP) which are greatly sped up on a GPU by exploiting massive parallelism. Specifically, ISGMR is an iterative and revised version of the standard SGM for general pairwise MRFs with improved optimization effectiveness, and TRWP is a highly parallel version of Sequential TRW (TRWS) for faster optimization. Our experiments on the standard stereo and denoising benchmarks demonstrated that ISGMR and TRWP achieve much lower energies than SGM and Mean-Field (MF), and TRWP is two orders of magnitude faster than TRWS without losing effectiveness in optimization. We further demonstrated the effectiveness of our algorithms on end-to-end learning for semantic segmentation. Notably, our CUDA implementations are at least 7 and 700 times faster than PyTorch GPU implementations for forward and backward propagation respectively, enabling efficient end-to-end learning with message passing.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://vision.middlebury.edu/MRF/results.

References

  1. Hirschmuller, H.: Stereo processing by semiglobal matching and mutual information. TPAMI 30, 328–341 (2008)

    Article  Google Scholar 

  2. Boykov, Y., Jolly, M.: Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images. In: ICCV (2011)

    Google Scholar 

  3. Hassner, M., Sklansky, J.: The use of Markov random fields as models of texture. Comput. Graph. Image Process. 12(4), 357–370 (1980)

    Google Scholar 

  4. Szeliski, R., et al.: A comparative study of energy minimization methods for Markov random fields with smoothness-based priors. TPAMI 30, 1068–1080 (2008)

    Article  Google Scholar 

  5. Zheng, S., et al.: Conditional random fields as recurrent neural networks. In: CVPR (2015)

    Google Scholar 

  6. Krähenbühl, P., Koltunz, V.: Efficient inference in fully connected CRFs with Gaussian edge potentials. In: NeurIPS (2011)

    Google Scholar 

  7. Seki, A., Pollefeys, M.: SGM-Nets: semi-global matching with neural networks. In: CVPR (2017)

    Google Scholar 

  8. Drory, A., Haubold, C., Avidan, S., Hamprecht, F.A.: Semi-global matching: a principled derivation in terms of message passing. In: Proceedings of German Conference on Pattern Recognition (GCPR) (2014)

    Google Scholar 

  9. Kolmogorov, V.: Convergent tree-reweighted message passing for energy minimization. TPAMI 28, 1568–1583 (2006)

    Article  Google Scholar 

  10. Ajanthan, T., Hartley, R., Salzmann, M.: Memory efficient max-flow for multi-label submodular MRFs. In: CVPR (2016)

    Google Scholar 

  11. Ajanthan, T., Hartley, R., Salzmann, M., Li, H.: Iteratively reweighted graph cut for multi-label MRFs with non-convex priors. In: CVPR (2015)

    Google Scholar 

  12. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. TPAMI 23, 1222–1239 (2001)

    Article  Google Scholar 

  13. Carr, P., Hartley, R.: Solving multilabel graph cut problems with multilabel swap. In: DICTA (2009)

    Google Scholar 

  14. Hartley, R., Ajanthan, T.: Generalized range moves. arXiv:1811.09171 (2018)

  15. Veksler, O.: Multi-label moves for MRFs with truncated convex priors. IJCV 98, 1–14 (2012). https://doi.org/10.1007/s11263-011-0491-6

    Article  MathSciNet  MATH  Google Scholar 

  16. Jordan, M.: Learning in Graphical Models. MIT Press, Cambridge (1998)

    Book  Google Scholar 

  17. Kwon, D., Lee, K.J., Yun, I.D., Lee, S.U.: Solving MRFs with higher-order smoothness priors using hierarchical gradient nodes. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010. LNCS, vol. 6492, pp. 121–134. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-19315-6_10

    Chapter  Google Scholar 

  18. Murphy, K., Weiss, Y., Jordan, M.: Loopy belief propagation for approximate inference: an empirical study. In: UAI (1999)

    Google Scholar 

  19. Pearl, J.: Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann, Burlington (1988)

    MATH  Google Scholar 

  20. Wainwright, M.J., Jordan, M.I.: Graphical models, exponential families, and variational inference. Found. Trends Mach. Learn. 1, 1–305 (2008)

    Article  Google Scholar 

  21. Wang, Z., Zhang, Z., Geng, N.: A message passing algorithm for MRF inference with unknown graphs and its applications. In: Cremers, D., Reid, I., Saito, H., Yang, M.-H. (eds.) ACCV 2014. LNCS, vol. 9006, pp. 288–302. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16817-3_19

    Chapter  Google Scholar 

  22. Kappes, J., et al.: A comparative study of modern inference techniques for discrete energy minimization problems. In: CVPR (2013)

    Google Scholar 

  23. Domke, J.: Learning graphical model parameters with approximate marginal inference. TPAMI 35, 2454–2467 (2013)

    Article  Google Scholar 

  24. Taskar, B., Guestrin, C., Koller, D.: Max-Margin Markov Networks. MIT Press, Cambridge (2003)

    Google Scholar 

  25. Tsochantaridis, I., Joachims, T., Hofmann, T., Altun, Y.: Large margin methods for structured and interdependent output variables. JMLR 6, 1453–1484 (2005)

    MathSciNet  MATH  Google Scholar 

  26. Liu, Z., Li, X., Luo, P., Loy, C.C., Tang, X.: Semantic image segmentation via deep parsing network. In: ICCV (2015)

    Google Scholar 

  27. Lin, G., Shen, C., Hengel, A., Reid, I.: Efficient piecewise training of deep structured models for semantic segmentation. In: CVPR (2016)

    Google Scholar 

  28. Zhang, F., Prisacariu, V., Yang, R., Torr, P.H.: GA-Net: guided aggregation net for end-to-end stereo matching. In: CVPR (2019)

    Google Scholar 

  29. Knobelreiter, P., Reinbacher, C., Shekhovtsov, A., Pock, T.: End-to-end training of hybrid CNN-CRF models for stereo. In: CVPR (2017)

    Google Scholar 

  30. Facciolo, G., Franchis, C., Meinhardt, E.: MGM: a significantly more global matching for stereo vision. In: BMVC (2015)

    Google Scholar 

  31. Hernandez-Juare, D., Chacon, A., Espinosa, A., Vazquez, D., Moure, J., Lopez, A.M.L.: Embedded real-time stereo estimation via semi-global matching on the GPU. In: International Conference on Computational Sciences (2016)

    Google Scholar 

  32. Wainwright, M., Jaakkola, T., Willsky, A.: MAP estimation via agreement on (hyper) trees: message-passing and linear-programming approaches. Trans. Inf. Theory 51(11), 3697–3717 (2005)

    Google Scholar 

  33. Dagum, L., Menon, R.: OpenMP: an industry standard API for shared-memory programming. Comput. Sci. Eng. 5, 46–55 (1998)

    Google Scholar 

  34. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Comput. Vis. 47, 7–42 (2002). https://doi.org/10.1023/A:1014573219977

    Article  MATH  Google Scholar 

  35. Scharstein, D., Szeliski, R.: High-accuracy stereo depth maps using structured light. In: CVPR (2003)

    Google Scholar 

  36. Menze, M., Heipke, C., Geiger, A.: Object scene flow. ISPRS J. Photogram. Remote Sens. 140, 60–76 (2018)

    Article  Google Scholar 

  37. Menze, M., Heipke, C., Geiger, A.: Joint 3D estimation of vehicles and scene flow. In: ISPRS Workshop on Image Sequence Analysis (2015)

    Google Scholar 

  38. Schops, T., et al.: A multi-view stereo benchmark with high-resolution images and multi-camera videos. In: CVPR (2017)

    Google Scholar 

  39. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2003)

    MATH  Google Scholar 

  40. Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with Atrous separable convolution for semantic image segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 833–851. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_49

    Chapter  Google Scholar 

  41. Everingham, M., Eslami, S., Gool, L., Williams, C., Winn, J., Zisserman, A.: The pascal visual object classes challenge a retrospective. Int. J. Comput. Vis. 111, 98–136 (2015). https://doi.org/10.1007/s11263-014-0733-5

    Article  Google Scholar 

  42. Hariharan, B., Arbelaez, P., Bourdev, L., Maji, S., Malik, J.: Semantic contours from inverse detectors. In: ICCV (2011)

    Google Scholar 

Download references

Acknowledgement

We would like to thank our colleagues Dylan Campbell and Yao Lu for the discussion of CUDA programming. This work is supported by the Australian Centre for Robotic Vision (CE140100016) and Data61, CSIRO, Canberra, Australia.

Author information

Authors and Affiliations

  1. Australian National University and Australian Centre for Robotic Vision, Canberra, Australia

    Zhiwei Xu, Thalaiyasingam Ajanthan & Richard Hartley

  2. Data61, CSIRO, Canberra, Australia

    Zhiwei Xu

Authors
  1. Zhiwei Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Thalaiyasingam Ajanthan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Richard Hartley
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhiwei Xu .

Editor information

Editors and Affiliations

  1. Waseda University, Tokyo, Japan

    Hiroshi Ishikawa

  2. Institute of Automation of Chinese Academy of Sciences, Beijing, China

    Cheng-Lin Liu

  3. Czech Technical University in Prague, Prague, Czech Republic

    Tomas Pajdla

  4. University of Pennsylvania, Philadelphia, PA, USA

    Jianbo Shi

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 934 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xu, Z., Ajanthan, T., Hartley, R. (2021). Fast and Differentiable Message Passing on Pairwise Markov Random Fields. In: Ishikawa, H., Liu, CL., Pajdla, T., Shi, J. (eds) Computer Vision – ACCV 2020. ACCV 2020. Lecture Notes in Computer Science(), vol 12624. Springer, Cham. https://doi.org/10.1007/978-3-030-69535-4_32

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-69535-4_32

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-69534-7

  • Online ISBN: 978-3-030-69535-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation