Skip to main content
SpringerLink
Log in

Learning Energy Based Inpainting for Optical Flow

  • Conference paper
  • First Online:
Computer Vision – ACCV 2018 (ACCV 2018)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 11366))

Included in the following conference series:

Abstract

Modern optical flow methods are often composed of a cascade of many independent steps or formulated as a black box neural network that is hard to interpret and analyze. In this work we seek for a plain, interpretable, but learnable solution. We propose a novel inpainting based algorithm that approaches the problem in three steps: feature selection and matching, selection of supporting points and energy based inpainting. To facilitate the inference we propose an optimization layer that allows to backpropagate through 10K iterations of a first-order method without any numerical or memory problems. Compared to recent state-of-the-art networks, our modular CNN is very lightweight and competitive with other, more involved, inpainting based methods.

Supported by the ERC starting grant 640156, ‘HOMOVIS’.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://github.com/vogechri/CustomNetworkLayers.

References

  1. Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34, 2274–2282 (2012)

    Article  Google Scholar 

  2. Agresti, G., Minto, L., Marin, G., Zanuttigh, P.: Deep learning for confidence information in stereo and tof data fusion. In: The IEEE International Conference on Computer Vision (ICCV) Workshops, October 2017

    Google Scholar 

  3. Barron, J.T., Poole, B.: The fast bilateral solver. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 617–632. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_38

    Chapter  Google Scholar 

  4. Beck, A., Teboulle, M.: SIAM J. Imaging Sci. A fast iterative shrinkage-thresholding algorithm for linear inverse problems 2, 183–202 (2009)

    Google Scholar 

  5. Bredies, K., Kunisch, K., Pock, T.: SIAM J. Imaging Sci. Total generalized variation 3, 492–526 (2010)

    Google Scholar 

  6. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24673-2_3

    Chapter  Google Scholar 

  7. Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 611–625. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33783-3_44

    Chapter  Google Scholar 

  8. Chen, T., Xu, B., Zhang, C., Guestrin, C.: Training deep nets with sublinear memory cost. CoRR abs/1604.06174 (2016)

    Google Scholar 

  9. Chen, Y., Ranftl, R., Pock, T.: A bi-level view of inpainting - based image compression. CoRR abs/1401.4112 (2014)

    Google Scholar 

  10. Dollár, P., Zitnick, C.L.: Structured forests for fast edge detection. In: CVPR, ICCV 2013, pp. 1841–1848. IEEE (2013)

    Google Scholar 

  11. Dosovitskiy, A., et al.: FlowNet: learning optical flow with convolutional networks. In: ICCV (2015)

    Google Scholar 

  12. Galić, I., Weickert, J., Welk, M., Bruhn, A., Belyaev, A., Seidel, H.P.: Image compression with anisotropic diffusion. J. Math. Imaging Vis. 31(2), 255–269 (2008). https://doi.org/10.1007/s10851-008-0087-0

    Article  MathSciNet  MATH  Google Scholar 

  13. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? In: CVPR (2012)

    Google Scholar 

  14. Griewank, A., Walther, A.: Algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of computational differentiation. ACM Trans. Math. Softw. 26(1), 19–45 (2000)

    Article  Google Scholar 

  15. Güney, F., Geiger, A.: Deep discrete flow. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10114, pp. 207–224. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-54190-7_13

    Chapter  Google Scholar 

  16. Hu, X., Mordohai, P.: A quantitative evaluation of confidence measures for stereo vision. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2121–2133 (2012)

    Article  Google Scholar 

  17. Hu, Y., Li, Y., Song, R.: Robust interpolation of correspondences for large displacement optical flow. In: CVPR. IEEE (2017)

    Google Scholar 

  18. Hu, Y., Song, R., Li, Y.: Efficient coarse-to-fine patchmatch for large displacement optical flow. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

    Google Scholar 

  19. Hu, Y., Song, R., Li, Y.: Efficient coarse-to-fine patchmatch for large displacement optical flow. In: CVPR. IEEE (2016)

    Google Scholar 

  20. Hui, T.W., Tang, X., Change Loy, C.: Liteflownet: a lightweight convolutional neural network for optical flow estimation. In: CVPR (2018)

    Google Scholar 

  21. Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: evolution of optical flow estimation with deep networks. In: CVPR (2017)

    Google Scholar 

  22. Menze, M., Geiger, A.: Object scene flow for autonomous vehicles. In: CVPR (2015)

    Google Scholar 

  23. Munda, G., Shekhovtsov, A., Knöbelreiter, P., Pock, T.: Scalable full flow with learned binary descriptors. In: Roth, V., Vetter, T. (eds.) GCPR 2017. LNCS, vol. 10496, pp. 321–332. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66709-6_26

    Chapter  Google Scholar 

  24. Nesterov, Y.: A method of solving a convex programming problem with convergence rate O(\(\frac{1}{k^2}\)). Sov. Math. Dokl. 27 (1983)

    Google Scholar 

  25. Peter, P., Hoffmann, S., Nedwed, F., Hoeltgen, L., Weickert, J.: From optimised inpainting with linear PDEs towards competitive image compression codecs. In: Bräunl, T., McCane, B., Rivera, M., Yu, X. (eds.) PSIVT 2015. LNCS, vol. 9431, pp. 63–74. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-29451-3_6

    Chapter  Google Scholar 

  26. Poggi, M., Tosi, F., Mattoccia, S.: Quantitative evaluation of confidence measures in a machine learning world. In: ICCV, October 2017

    Google Scholar 

  27. Ranjan, A., Black, M.J.: Optical flow estimation using a spatial pyramid network. In: CVPR (2017)

    Google Scholar 

  28. Revaud, J., Weinzaepfel, P., Harchaoui, Z., Schmid, C.: Epicflow: edge-preserving interpolation of correspondences for optical flow. In: CVPR. IEEE (2015)

    Google Scholar 

  29. Rhemann, C., Hosni, A., Bleyer, M., Rother, C., Gelautz, M.: Fast cost-volume filtering for visual correspondence and beyond. In: CVPR, pp. 3017–3024. IEEE (2011)

    Google Scholar 

  30. Riegler, G., Rüther, M., Bischof, H.: ATGV-Net: accurate depth super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 268–284. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_17

    Chapter  Google Scholar 

  31. Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D Nonlinear Phenom. 60, 259–268 (1992)

    Article  MathSciNet  Google Scholar 

  32. Sun, D., Yang, X., Liu, M.Y., Kautz, J.: PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In: CVPR (2018)

    Google Scholar 

  33. Theano Development Team: Theano: a Python framework for fast computation of mathematical expressions. CoRR (2016)

    Google Scholar 

  34. Vogel, C., Pock, T.: A primal dual network for low-level vision problems. In: Roth, V., Vetter, T. (eds.) GCPR 2017. LNCS, vol. 10496, pp. 189–202. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66709-6_16

    Chapter  Google Scholar 

  35. Wang, S., Fanello, S.R., Rhemann, C., Izadi, S., Kohli, P.: The global patch collider. In: CVPR. IEEE, July 2016

    Google Scholar 

  36. Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: DeepFlow: large displacement optical flow with deep matching. In: ICCV. IEEE, Sydney, December 2013

    Google Scholar 

  37. Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., Bischof, H.: Anisotropic Huber-L1 optical flow. In: BMVC (2009)

    Google Scholar 

  38. Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. In: ICLR (2016)

    Google Scholar 

  39. Zheng, S., et al.: Conditional random fields as recurrent neural networks. In: ICCV (2015)

    Google Scholar 

  40. Zweig, S., Wolf, L.: InterpoNet, a brain inspired neural network for optical flow dense interpolation. In: CVPR, pp. 6363–6372 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Graz University of Technology, Graz, Austria

    Christoph Vogel, Patrick Knöbelreiter & Thomas Pock

  2. Austrian Institute of Technology, Vienna, Austria

    Thomas Pock

Authors
  1. Christoph Vogel
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Patrick Knöbelreiter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas Pock
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Christoph Vogel .

Editor information

Editors and Affiliations

  1. IIIT Hyderabad, Hyderabad, India

    C.V. Jawahar

  2. ANU, Canberra, ACT, Australia

    Hongdong Li

  3. Simon Fraser University, Burnaby, BC, Canada

    Greg Mori

  4. ETH Zurich, Zurich, Zürich, Switzerland

    Konrad Schindler

1 Electronic supplementary material

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Vogel, C., Knöbelreiter, P., Pock, T. (2019). Learning Energy Based Inpainting for Optical Flow. In: Jawahar, C., Li, H., Mori, G., Schindler, K. (eds) Computer Vision – ACCV 2018. ACCV 2018. Lecture Notes in Computer Science(), vol 11366. Springer, Cham. https://doi.org/10.1007/978-3-030-20876-9_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-20876-9_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-20875-2

  • Online ISBN: 978-3-030-20876-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation