Learning Energy Based Inpainting for Optical Flow

  • Christoph VogelEmail author
  • Patrick Knöbelreiter
  • Thomas Pock
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11366)


Modern optical flow methods are often composed of a cascade of many independent steps or formulated as a black box neural network that is hard to interpret and analyze. In this work we seek for a plain, interpretable, but learnable solution. We propose a novel inpainting based algorithm that approaches the problem in three steps: feature selection and matching, selection of supporting points and energy based inpainting. To facilitate the inference we propose an optimization layer that allows to backpropagate through 10K iterations of a first-order method without any numerical or memory problems. Compared to recent state-of-the-art networks, our modular CNN is very lightweight and competitive with other, more involved, inpainting based methods.


Optical flow Energy optimization Deep learning 

Supplementary material

484523_1_En_22_MOESM1_ESM.pdf (169 kb)
Supplementary material 1 (pdf 168 KB)
484523_1_En_22_MOESM2_ESM.mp4 (15.5 mb)
Supplementary material 2 (mp4 15835 KB)


  1. 1.
    Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34, 2274–2282 (2012)CrossRefGoogle Scholar
  2. 2.
    Agresti, G., Minto, L., Marin, G., Zanuttigh, P.: Deep learning for confidence information in stereo and tof data fusion. In: The IEEE International Conference on Computer Vision (ICCV) Workshops, October 2017Google Scholar
  3. 3.
    Barron, J.T., Poole, B.: The fast bilateral solver. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 617–632. Springer, Cham (2016). Scholar
  4. 4.
    Beck, A., Teboulle, M.: SIAM J. Imaging Sci. A fast iterative shrinkage-thresholding algorithm for linear inverse problems 2, 183–202 (2009)Google Scholar
  5. 5.
    Bredies, K., Kunisch, K., Pock, T.: SIAM J. Imaging Sci. Total generalized variation 3, 492–526 (2010)Google Scholar
  6. 6.
    Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004). Scholar
  7. 7.
    Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 611–625. Springer, Heidelberg (2012). Scholar
  8. 8.
    Chen, T., Xu, B., Zhang, C., Guestrin, C.: Training deep nets with sublinear memory cost. CoRR abs/1604.06174 (2016)Google Scholar
  9. 9.
    Chen, Y., Ranftl, R., Pock, T.: A bi-level view of inpainting - based image compression. CoRR abs/1401.4112 (2014)Google Scholar
  10. 10.
    Dollár, P., Zitnick, C.L.: Structured forests for fast edge detection. In: CVPR, ICCV 2013, pp. 1841–1848. IEEE (2013)Google Scholar
  11. 11.
    Dosovitskiy, A., et al.: FlowNet: learning optical flow with convolutional networks. In: ICCV (2015)Google Scholar
  12. 12.
    Galić, I., Weickert, J., Welk, M., Bruhn, A., Belyaev, A., Seidel, H.P.: Image compression with anisotropic diffusion. J. Math. Imaging Vis. 31(2), 255–269 (2008). Scholar
  13. 13.
    Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? In: CVPR (2012)Google Scholar
  14. 14.
    Griewank, A., Walther, A.: Algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of computational differentiation. ACM Trans. Math. Softw. 26(1), 19–45 (2000)CrossRefGoogle Scholar
  15. 15.
    Güney, F., Geiger, A.: Deep discrete flow. In: Lai, S.-H., Lepetit, V., Nishino, K., Sato, Y. (eds.) ACCV 2016. LNCS, vol. 10114, pp. 207–224. Springer, Cham (2017). Scholar
  16. 16.
    Hu, X., Mordohai, P.: A quantitative evaluation of confidence measures for stereo vision. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2121–2133 (2012)CrossRefGoogle Scholar
  17. 17.
    Hu, Y., Li, Y., Song, R.: Robust interpolation of correspondences for large displacement optical flow. In: CVPR. IEEE (2017)Google Scholar
  18. 18.
    Hu, Y., Song, R., Li, Y.: Efficient coarse-to-fine patchmatch for large displacement optical flow. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016Google Scholar
  19. 19.
    Hu, Y., Song, R., Li, Y.: Efficient coarse-to-fine patchmatch for large displacement optical flow. In: CVPR. IEEE (2016)Google Scholar
  20. 20.
    Hui, T.W., Tang, X., Change Loy, C.: Liteflownet: a lightweight convolutional neural network for optical flow estimation. In: CVPR (2018)Google Scholar
  21. 21.
    Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: evolution of optical flow estimation with deep networks. In: CVPR (2017)Google Scholar
  22. 22.
    Menze, M., Geiger, A.: Object scene flow for autonomous vehicles. In: CVPR (2015)Google Scholar
  23. 23.
    Munda, G., Shekhovtsov, A., Knöbelreiter, P., Pock, T.: Scalable full flow with learned binary descriptors. In: Roth, V., Vetter, T. (eds.) GCPR 2017. LNCS, vol. 10496, pp. 321–332. Springer, Cham (2017). Scholar
  24. 24.
    Nesterov, Y.: A method of solving a convex programming problem with convergence rate O(\(\frac{1}{k^2}\)). Sov. Math. Dokl. 27 (1983)Google Scholar
  25. 25.
    Peter, P., Hoffmann, S., Nedwed, F., Hoeltgen, L., Weickert, J.: From optimised inpainting with linear PDEs towards competitive image compression codecs. In: Bräunl, T., McCane, B., Rivera, M., Yu, X. (eds.) PSIVT 2015. LNCS, vol. 9431, pp. 63–74. Springer, Cham (2016). Scholar
  26. 26.
    Poggi, M., Tosi, F., Mattoccia, S.: Quantitative evaluation of confidence measures in a machine learning world. In: ICCV, October 2017Google Scholar
  27. 27.
    Ranjan, A., Black, M.J.: Optical flow estimation using a spatial pyramid network. In: CVPR (2017)Google Scholar
  28. 28.
    Revaud, J., Weinzaepfel, P., Harchaoui, Z., Schmid, C.: Epicflow: edge-preserving interpolation of correspondences for optical flow. In: CVPR. IEEE (2015)Google Scholar
  29. 29.
    Rhemann, C., Hosni, A., Bleyer, M., Rother, C., Gelautz, M.: Fast cost-volume filtering for visual correspondence and beyond. In: CVPR, pp. 3017–3024. IEEE (2011)Google Scholar
  30. 30.
    Riegler, G., Rüther, M., Bischof, H.: ATGV-Net: accurate depth super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 268–284. Springer, Cham (2016). Scholar
  31. 31.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Phys. D Nonlinear Phenom. 60, 259–268 (1992)MathSciNetCrossRefGoogle Scholar
  32. 32.
    Sun, D., Yang, X., Liu, M.Y., Kautz, J.: PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In: CVPR (2018)Google Scholar
  33. 33.
    Theano Development Team: Theano: a Python framework for fast computation of mathematical expressions. CoRR (2016)Google Scholar
  34. 34.
    Vogel, C., Pock, T.: A primal dual network for low-level vision problems. In: Roth, V., Vetter, T. (eds.) GCPR 2017. LNCS, vol. 10496, pp. 189–202. Springer, Cham (2017). Scholar
  35. 35.
    Wang, S., Fanello, S.R., Rhemann, C., Izadi, S., Kohli, P.: The global patch collider. In: CVPR. IEEE, July 2016Google Scholar
  36. 36.
    Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: DeepFlow: large displacement optical flow with deep matching. In: ICCV. IEEE, Sydney, December 2013Google Scholar
  37. 37.
    Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., Bischof, H.: Anisotropic Huber-L1 optical flow. In: BMVC (2009)Google Scholar
  38. 38.
    Yu, F., Koltun, V.: Multi-scale context aggregation by dilated convolutions. In: ICLR (2016)Google Scholar
  39. 39.
    Zheng, S., et al.: Conditional random fields as recurrent neural networks. In: ICCV (2015)Google Scholar
  40. 40.
    Zweig, S., Wolf, L.: InterpoNet, a brain inspired neural network for optical flow dense interpolation. In: CVPR, pp. 6363–6372 (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Graz University of TechnologyGrazAustria
  2. 2.Austrian Institute of TechnologyViennaAustria

Personalised recommendations