Learning Optical Flow

  • Deqing Sun
  • Stefan Roth
  • J. P. Lewis
  • Michael J. Black
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5304)


Assumptions of brightness constancy and spatial smoothness underlie most optical flow estimation methods. In contrast to standard heuristic formulations, we learn a statistical model of both brightness constancy error and the spatial properties of optical flow using image sequences with associated ground truth flow fields. The result is a complete probabilistic model of optical flow. Specifically, the ground truth enables us to model how the assumption of brightness constancy is violated in naturalistic sequences, resulting in a probabilistic model of “brightness inconstancy”. We also generalize previous high-order constancy assumptions, such as gradient constancy, by modeling the constancy of responses to various linear filters in a high-order random field framework. These filters are free variables that can be learned from training data. Additionally we study the spatial structure of the optical flow and how motion boundaries are related to image intensity boundaries. Spatial smoothness is modeled using a Steerable Random Field, where spatial derivatives of the optical flow are steered by the image brightness structure. These models provide a statistical motivation for previous methods and enable the learning of all parameters from training data. All proposed models are quantitatively compared on the Middlebury flow dataset.


Ground Truth Optical Flow Data Term Spatial Term Brightness Constancy 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Black, M.J., Anandan, P.: The robust estimation of multiple motions: Parametric and piecewise-smooth flow fields. CVIU 63, 75–104 (1996)Google Scholar
  2. 2.
    Roth, S., Black, M.J.: On the spatial statistics of optical flow. IJCV 74, 33–50 (2007)CrossRefGoogle Scholar
  3. 3.
    Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., Szeliski, R.: A database and evaluation methodology for optical flow. In: ICCV (2007)Google Scholar
  4. 4.
    Nagel, H.H., Enkelmann, W.: An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE TPAMI 8, 565–593 (1986)CrossRefGoogle Scholar
  5. 5.
    Roth, S., Black, M.J.: Steerable random fields. In: ICCV (2007)Google Scholar
  6. 6.
    Wainwright, M.J., Simoncelli, E.P.: Scale mixtures of Gaussians and the statistics of natural images. In: NIPS, pp. 855–861 (1999)Google Scholar
  7. 7.
    Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J(G.) (eds.) ECCV 2004. LNCS, vol. 3024, pp. 25–36. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  8. 8.
    Bruhn, A., Weickert, J., Schnörr, C.: Lucas/Kanade meets Horn/Schunck: combining local and global optic flow methods. IJCV 61, 211–231 (2005)CrossRefGoogle Scholar
  9. 9.
    Horn, B., Schunck, B.: Determining optical flow. Artificial Intelligence 16, 185–203 (1981)CrossRefGoogle Scholar
  10. 10.
    Fermüller, C., Shulman, D., Aloimonos, Y.: The statistics of optical flow. CVIU 82, 1–32 (2001)zbMATHGoogle Scholar
  11. 11.
    Gennert, M.A., Negahdaripour, S.: Relaxing the brightness constancy assumption in computing optical flow. Technical report, Cambridge, MA, USA (1987)Google Scholar
  12. 12.
    Haussecker, H., Fleet, D.: Computing optical flow with physical models of brightness variation. IEEE TPAMI 23, 661–673 (2001)CrossRefGoogle Scholar
  13. 13.
    Toth, D., Aach, T., Metzler, V.: Illumination-invariant change detection. In: 4th IEEE Southwest Symposium on Image Analysis and Interpretation, pp. 3–7 (2000)Google Scholar
  14. 14.
    Adelson, E.H., Anderson, C.H., Bergen, J.R., Burt, P.J., Ogden, J.M.: Pyramid methods in image processing. RCA Engineer 29, 33–41 (1984)Google Scholar
  15. 15.
    Alvarez, L., Deriche, R., Papadopoulo, T., Sanchez, J.: Symmetrical dense optical flow estimation with occlusions detection. IJCV 75, 371–385 (2007)CrossRefzbMATHGoogle Scholar
  16. 16.
    Fleet, D.J., Black, M.J., Nestares, O.: Bayesian inference of visual motion boundaries. In: Exploring Artificial Intelligence in the New Millennium, pp. 139–174. Morgan Kaufmann Pub., San Francisco (2002)Google Scholar
  17. 17.
    Black, M.J.: Combining intensity and motion for incremental segmentation and tracking over long image sequences. In: Sandini, G. (ed.) ECCV 1992. LNCS, vol. 588, pp. 485–493. Springer, Heidelberg (1992)CrossRefGoogle Scholar
  18. 18.
    Simoncelli, E.P., Adelson, E.H., Heeger, D.J.: Probability distributions of optical flow. In: CVPR, pp. 310–315 (1991)Google Scholar
  19. 19.
    Black, M.J., Yacoob, Y., Jepson, A.D., Fleet, D.J.: Learning parameterized models of image motion. In: CVPR, pp. 561–567 (1997)Google Scholar
  20. 20.
    Freeman, W.T., Pasztor, E.C., Carmichael, O.T.: Learning low-level vision. IJCV 40, 25–47 (2000)CrossRefzbMATHGoogle Scholar
  21. 21.
    Scharstein, D., Pal, C.: Learning conditional random fields for stereo. In: CVPR (2007)Google Scholar
  22. 22.
    Roth, S., Black, M.J.: Fields of experts: A framework for learning image priors. In: CVPR, vol. II, pp. 860–867 (2005)Google Scholar
  23. 23.
    Stewart, L., He, X., Zemel, R.: Learning flexible features for conditional random fields. IEEE TPAMI 30, 1145–1426 (2008)CrossRefGoogle Scholar
  24. 24.
    Hinton, G.E.: Training products of experts by minimizing contrastive divergence. Neural Comput 14, 1771–1800 (2002)CrossRefzbMATHGoogle Scholar
  25. 25.
    Blake, A., Zisserman, A.: Visual Reconstruction. The MIT Press, Cambridge, Massachusetts (1987)Google Scholar
  26. 26.
    Zhu, S., Wu, Y., Mumford, D.: Filters random fields and maximum entropy (FRAME): To a unified theory for texture modeling. IJCV 27, 107–126 (1998)CrossRefGoogle Scholar
  27. 27.
    Lempitsky, V., Roth, S., Rother, C.: FusionFlow: Discrete-continuous optimization for optical flow estimation. In: CVPR (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Deqing Sun
    • 1
  • Stefan Roth
    • 2
  • J. P. Lewis
    • 3
  • Michael J. Black
    • 1
  1. 1.Department of Computer ScienceBrown UniversityProvidenceUSA
  2. 2.Department of Computer ScienceTU DarmstadtDarmstadtGermany
  3. 3.Weta Digital Ltd.New Zealand

Personalised recommendations