Efficient Nonlocal Regularization for Optical Flow

  • Philipp Krähenbühl
  • Vladlen Koltun
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7572)


Dense optical flow estimation in images is a challenging problem because the algorithm must coordinate the estimated motion across large regions in the image, while avoiding inappropriate smoothing over motion boundaries. Recent works have advocated for the use of nonlocal regularization to model long-range correlations in the flow. However, incorporating nonlocal regularization into an energy optimization framework is challenging due to the large number of pairwise penalty terms. Existing techniques either substitute intermediate filtering of the flow field for direct optimization of the nonlocal objective, or suffer substantial performance penalties when the range of the regularizer increases. In this paper, we describe an optimization algorithm that efficiently handles a general type of nonlocal regularization objectives for optical flow estimation. The computational complexity of the algorithm is independent of the range of the regularizer. We show that nonlocal regularization improves estimation accuracy at longer ranges than previously reported, and is complementary to intermediate filtering of the flow field. Our algorithm is simple and is compatible with many optical flow models.


Optical Flow Penalty Function Regularization Term Data Term Angular Error 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Adams, A., Baek, J., Davis, M.A.: Fast high-dimensional filtering using the permutohedral lattice. Computer Graphics Forum 29(2) (2010)Google Scholar
  2. 2.
    Andrews, D.F., Mallows, C.L.: Scale mixtures of normal distributions. Journal of the Royal Statistical Society. Series B (Methodological) 36(1), 99–102 (1974)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Aujol, J.-F., Gilboa, G., Chan, T., Osher, S.: Structure-texture image decomposition–modeling, algorithms, and parameter selection. International Journal of Computer Vision 67, 111–136 (2006)CrossRefGoogle Scholar
  4. 4.
    Baker, S., Scharstein, D., Lewis, J.P., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. International Journal of Computer Vision 92(1), 1–31 (2011)CrossRefGoogle Scholar
  5. 5.
    Black, M.J., Anandan, P.: The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding 63(1), 75–104 (1996)CrossRefGoogle Scholar
  6. 6.
    Blake, A., Zisserman, A.: Visual Reconstruction. MIT Press (1987)Google Scholar
  7. 7.
    Bruhn, A., Weickert, J., Schnörr, C.: Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. International Journal of Computer Vision 61(3), 211–231 (2005)CrossRefGoogle Scholar
  8. 8.
    Horn, B.K.P., Schunck, B.G.: Determining optical flow. Artificial Intelligence 17, 185–203 (1981)CrossRefGoogle Scholar
  9. 9.
    Krähenbühl, P., Koltun, V.: Efficient inference in fully connected CRFs with Gaussian edge potentials. In: Proc. NIPS (2011)Google Scholar
  10. 10.
    Nagel, H.-H., Enkelmann, W.: An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence 8(5), 565–593 (1986)CrossRefGoogle Scholar
  11. 11.
    Papenberg, N., Bruhn, A., Brox, T., Didas, S., Weickert, J.: Highly accurate optic flow computation with theoretically justified warping. International Journal of Computer Vision 67(2), 141–158 (2006)CrossRefGoogle Scholar
  12. 12.
    Paris, S., Durand, F.: A fast approximation of the bilateral filter using a signal processing approach. International Journal of Computer Vision 81(1), 24–52 (2009)CrossRefGoogle Scholar
  13. 13.
    Portilla, J., Strela, V., Wainwright, M.J., Simoncelli, E.P.: Image denoising using scale mixtures of Gaussians in the wavelet domain. IEEE Transactions on Image Processing 12(11), 1338–1351 (2003)MathSciNetCrossRefGoogle Scholar
  14. 14.
    Press, W.H., Teukolsky, S.A., Vetterling, W.T., Flannery, B.P.: Numerical Recipes: The Art of Scientific Computing, 3rd edn. Cambridge University Press (2007)Google Scholar
  15. 15.
    Roth, S., Black, M.J.: On the spatial statistics of optical flow. International Journal of Computer Vision 74(1), 33–50 (2007)CrossRefGoogle Scholar
  16. 16.
    Rudin, L.I., Osher, S., Fatemi, E.: Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena 60(1-4), 259–268 (1992)zbMATHCrossRefGoogle Scholar
  17. 17.
    Steinbruecker, F., Pock, T., Cremers, D.: Advanced data terms for variational optic flow estimation. In: Proc. VMV (2009)Google Scholar
  18. 18.
    Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: Proc. CVPR (2010)Google Scholar
  19. 19.
    Sun, D., Roth, S., Lewis, J.P., Black, M.J.: Learning Optical Flow. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part III. LNCS, vol. 5304, pp. 83–97. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  20. 20.
    Sun, D., Sudderth, E.B., Black, M.J.: Layered image motion with explicit occlusions, temporal consistency, and depth ordering. In: Proc. NIPS (2010)Google Scholar
  21. 21.
    Wedel, A., Pock, T., Zach, C., Cremers, D., Bischof, H.: An improved algorithm for TV-L 1 optical flow. In: Proc. of the Dagstuhl Motion Workshop. Springer (2008)Google Scholar
  22. 22.
    Weickert, J., Brox, T.: Diffusion and regularization of vector- and matrix-valued images. In: Inverse Problems, Image Analysis, and Medical Imaging, pp. 251–268. AMS (2002)Google Scholar
  23. 23.
    Werlberger, M., Pock, T., Bischof, H.: Motion estimation with non-local total variation regularization. In: Proc. CVPR (2010)Google Scholar
  24. 24.
    Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., Bischof, H.: Anisotropic Huber-L1 optical flow. In: Proc. BMVC (2009)Google Scholar
  25. 25.
    Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L 1 optical flow. In: DAGM-Symposium (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Philipp Krähenbühl
    • 1
  • Vladlen Koltun
    • 1
  1. 1.Stanford UniversityUSA

Personalised recommendations