Skip to main content
SpringerLink
Log in

Learning the Regularization Operator for the Optical Flow Problem

  • Published:
Programming and Computer Software Aims and scope Submit manuscript

Abstract

The task of displacement estimation for frames of a video sequence is considered. A new convolutional neural network architecture for the optical flow problem is proposed. The method is based on learning the regularization operator for a fast optimization method. The proposed method has low computational complexity and memory footprint at test time. The neural network architecture is based on unrolling iterations of a fast primal-dual method as layers of a convolutional neural network. Iterations of the optimization method are represented as convolutions with filters that are trained on ground truth data by backpropagation. A real-time implementation using graphics processing units is proposed. Experimental results demonstrate an improved quality of the optical flow field as compared to the optimization method based on a fixed regularization operator.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Horn, B.K.P. and Schunck, B.G., Determining optical flow, Artif. Intell., 1981, vol. 17, nos. 1–3, pp. 185–203.

    Article  Google Scholar 

  2. Sun, D., Roth, S., and Black, M.J., Secrets of optical flow estimation and their principles, Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2010, pp. 2432–2439

    Google Scholar 

  3. Chen, Q. and Koltun, V., Full flow: Optical flow estimation by global optimization over regular grids, Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4706–4714

    Google Scholar 

  4. Menze, M., Heipke, C., and Geiger, A., Discrete optimization for optical flow, Proc. German Conf. Pattern Recognition, 2015, pp. 16–28

    Google Scholar 

  5. Güney, F., Geiger, A., Deep discrete flow, Proc. Asian Conf. Computer Vision, 2016, pp. 207–224

    Google Scholar 

  6. Weinzaepfel, Ph., Revaud, J., Harchaoui, Z., et al., DeepFlow: Large displacement optical flow with deep matching, Proc. IEEE Int. Conf. Computer Vision, 2013, pp. 1385–1392

    Google Scholar 

  7. Hu, Y., Li, Y., and Song, R., Robust interpolation of correspondences for large displacement optical flow, Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2017.

    Google Scholar 

  8. Zweig, Sh. and Wolf, L., InterpoNet: A brain inspired neural network for optical flow dense interpolation, Preprint no. 1611.09803, 2016.

    Google Scholar 

  9. Revaud, J., Weinzaepfel, Ph., Harchaoui, Z., et al., Epicflow: Edge-preserving interpolation of correspondences for optical flow, Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1164–1172

    Google Scholar 

  10. Min Ba., Wenjie Luo, Kaustav Kundu, et al., Exploiting semantic information and deep matching for optical flow, Proc. European Conf. Computer Vision, 2016, pp. 154–170

    Google Scholar 

  11. Sevilla-Lara, L., Sun, D., Jampani, V., et al., Optical flow with semantic segmentation and localized layers, Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2016, pp. 3889–3898

    Google Scholar 

  12. Hur, J. and Roth, S., Joint optical flow and temporally consistent semantic segmentation, Proc. European Conf. Computer Vision, 2016, pp. 163–177

    Google Scholar 

  13. Dosovitskiy, A., Fischer, Ph., Ilg, E., et al., Flownet: Learning optical flow with convolutional networks, Proc. IEEE Int. Conf. Computer Vision, 2015, pp. 2758–2766

    Google Scholar 

  14. Ilg, E., Mayer, N., Saikia, T., et al., Flownet 2.0: Evolution of optical flow estimation with deep networks, Preprint no. 1612.01925, 2016.

    Google Scholar 

  15. Xu, J., Ranftl, R., and Koltun, V., Accurate optical flow via direct cost volume processing, Preprint no. 1704. 07325, 2017.

    Google Scholar 

  16. Rudin, L.I., Osher, S., and Fatemi, E., Nonlinear total variation based noise removal algorithms, Phys. D (Amsterdam, Neth.), 1992, vol. 60, nos. 1–4, pp. 259–268.

    Article  MathSciNet  MATH  Google Scholar 

  17. Zach, Ch., Pock, Th., and Bischof, H., A duality based approach for realtime TV-L1 optical flow, Pattern Recognit., 2007, pp. 214–223

    Chapter  Google Scholar 

  18. Wedel, A., Pock, Th., Zach, Ch., et al., An improved algorithm for TV-L1 optical flow, Statistical and Geometrical Approaches to Visual Motion Analysis, Springer, 2009, pp. 23–45

    Google Scholar 

  19. Bredies, K., Kunisch, K., and Pock, Th., Total generalized variation, SIAM J. Image Sci., 2010, vol. 3, no. 3, pp. 492–526.

    Article  MathSciNet  MATH  Google Scholar 

  20. Knoll, F., Bredies, K., Pock, Th., et al., Second order total generalized variation (TGV) for MRI, Magn. Reson. Med., 2011, vol. 65, no. 2, pp. 480–491.

    Article  Google Scholar 

  21. Ranftl, R., Gehrig, S., Pock, Th., et al., Pushing the limits of stereo using variational stereo estimation, Proc IEEE Intelligent Vehicles Symp. (IV), 2012, pp. 401–407

    Google Scholar 

  22. Werlberger, M., Convex approaches for high performance video processing, 2012.

    Google Scholar 

  23. Werlberger, M., Pock, Th., and Bischof, H., Motion estimation with non-local total variation regularization, Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2010, pp. 2464–2471

    Google Scholar 

  24. Ranftl, R., Bredies, K., and Pock, Th., Non-local total generalized variation for optical flow estimation, Proc. European Conf. Computer Vision, 2014, pp. 439–454

    Google Scholar 

  25. Steinbrucker, F., Pock, Th., and Cremers, D., Large displacement optical flow computation without warping, Proc. 12th IEEE Int. Conf. Computer Vision, 2009, pp. 1609–1614

    Google Scholar 

  26. Schmidt, U. and Roth, S., Shrinkage fields for effective image restoration, Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2014, pp. 2774–2781

    Google Scholar 

  27. Lefkimmiatis Stamatios, Non-local color image denoising with convolutional neural networks, Preprint no. 1611.06757, 2016.

    Google Scholar 

  28. Roth, S. and Black, M.J., Fields of experts: A framework for learning image priors, Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2005, vol. 2, pp. 860–867.

    Google Scholar 

  29. Wang, Sh., Fidler, S., and Urtasun, R., Proximal deep structured models, Advances in Neural Information Processing Systems, 2016, pp. 865–873

    Google Scholar 

  30. Jaderberg, M., Simonyan, K., Zisserman, A., et al., Spatial transformer networks, Advances in Neural Information Processing Systems, 2015, pp. 2017–2025

    Google Scholar 

  31. Butler, D.J., Wulff, J., Stanley, G.B., et al., A naturalistic open source movie for optical flow evaluation, Proc. European Conf. Computer Vision, 2012. pp. 611–625.

    Google Scholar 

  32. Kingma, D. and Ba, J., Adam: A method for stochastic optimization, Preprint no. 1412.6980, 2014.

    Google Scholar 

  33. Brox, Th. and Malik, J., Large displacement optical flow: Descriptor matching in variational motion estimation, IEEE Trans. Pattern Anal. Mach. Intell., 2011, vol. 33, no. 3, pp. 500–513.

    Article  Google Scholar 

  34. Sundaram, N., Brox, Th., and Keutzer, K., Dense point trajectories by GPU-accelerated large displacement optical flow, Proc. European Conf. Computer Vision, 2010, pp. 438–451

    Google Scholar 

  35. Revaud, J., Weinzaepfel, Ph., Harchaoui, Z., et al., Deepmatching: Hierarchical deformable dense matching, Int. J. Comput. Vision, 2016, vol. 120, no. 3, pp. 300–323.

    Article  MathSciNet  Google Scholar 

  36. Meister, S., Hur, J., and Roth, S., UnFlow: Unsupervised learning of optical flow with a bidirectional census loss, Preprint no. 1711.07837, 2017.

    Google Scholar 

  37. Bao Linchao, Yang Qingxion., and Jin Hailin, Fast edge-preserving patchmatch for large displacement optical flow, Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2014, pp. 3534–3541

    Google Scholar 

  38. Brox, Th., Bruhn, A., Papenberg, N., et al., High accuracy optical flow estimation based on a theory for warping, Comput. Vision ECCV, 2004, pp. 25–36

    Google Scholar 

  39. Linchao Bao, Hailin Jin, Byungmoon Kim, et al., A comparison of TV-L1 optical flow solvers on GPU, Proc. GPU Technol. Conf. (GTC) Posters, 2014, pp. 42–54

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Skolkolvo Institute of Science and Technology, ul. Nobelya 3, Moscow, 143026, Russia

    A. I. Kuzmin

Authors
  1. A. I. Kuzmin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to A. I. Kuzmin.

Additional information

Original Russian Text © A.I. Kuzmin, 2018, published in Programmirovanie, 2018, Vol. 44, No. 3.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kuzmin, A.I. Learning the Regularization Operator for the Optical Flow Problem. Program Comput Soft 44, 139–147 (2018). https://doi.org/10.1134/S0361768818030040

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1134/S0361768818030040

Search

Navigation