Journal of Mathematical Imaging and Vision

, Volume 14, Issue 3, pp 245–255 | Cite as

Variational Optic Flow Computation with a Spatio-Temporal Smoothness Constraint

  • Joachim Weickert
  • Christoph Schnörr


Nonquadratic variational regularization is a well-known and powerful approach for the discontinuity-preserving computation of optic flow. In the present paper, we consider an extension of flow-driven spatial smoothness terms to spatio-temporal regularizers. Our method leads to a rotationally invariant and time symmetric convex optimization problem. It has a unique minimum that can be found in a stable way by standard algorithms such as gradient descent. Since the convexity guarantees global convergence, the result does not depend on the flow initialization. Two iterative algorithms are presented that are not difficult to implement. Qualitative and quantitative results for synthetic and real-world scenes show that our spatio-temporal approach (i) improves optic flow fields significantly, (ii) smoothes out background noise efficiently, and (iii) preserves true motion boundaries. The computational costs are only 50% higher than for a pure spatial approach applied to all subsequent image pairs of the sequence.

optic flow differential techniques variational methods spatio-temporal regularization 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    L. Alvarez, J. Esclarín, M. Lefébure, and J. Sánchez, “A PDE model for computing the optical flow,” in Proc. XVI Congreso de Ecuaciones Diferenciales y Aplicaciones (C.E.D.Y.A. XVI, Las Palmas de Gran Canaria, Sept. 21–24, 1999), pp. 1349–1356, 1999.Google Scholar
  2. 2.
    L. Alvarez, J. Weickert, and J. Sánchez, “Reliable estimation of dense optical flow fields with large displacements,” Int. J. Comput. Vision, Vol. 39, pp. 41–56, 2000.Google Scholar
  3. 3.
    J. Arnspang, “Notes on local determination of smooth optic flow and the translational property of first order optic flow,” Report DIKU-88/1, Dept. of Computer Science, University of Copenhagen, Denmark, 1988.Google Scholar
  4. 4.
    G. Aubert, R. Deriche, and P. Kornprobst, “Computing optical flow via variational techniques,” SIAM J. Appl. Math., Vol. 60, pp. 156–182, 1999.Google Scholar
  5. 5.
    J.L. Barron, D.J. Fleet, and S.S. Beauchemin, “Performance of optical flow techniques,” Int. J. Comput. Vision, Vol. 12, pp. 43–77, 1994.Google Scholar
  6. 6.
    M.J. Black, “Recursive non-linear estimation of discontinuous flow fields,” in Computer Vision—ECCV '94, Volume I, Lecture Notes in Computer Science, Vol. 800, J.-O. Eklundh (Ed.), Springer: Berlin, 1994, pp. 138–145.Google Scholar
  7. 7.
    M.J. Black and P. Anandan, “Robust dynamic motion estimation over time,” in Proc. IEEE Comp. Soc. Conf. on Computer Vision and Pattern Recognition (CVPR '91, Maui, June 3–6, 1991), IEEE Computer Society Press: Los Alamitos, 1991, pp. 292–302.Google Scholar
  8. 8.
    A. Blake and A. Zisserman, Visual Reconstruction, MIT Press: Cambridge, Mass., 1987.Google Scholar
  9. 9.
    P. Charbonnier, “Reconstruction d'image: Régularisation avec prise en compte des discontinuités,” Ph.D. Thesis, Laboratoire Informatique, Signaux et Systémes de Sophia Antipolis, Université Nice—Sophia Antipolis, France, 1994.Google Scholar
  10. 10.
    P. Charbonnier, L. Blanc-Féraud, G. Aubert, and M. Barlaud, “Deterministic edge-preserving regularization in computed imaging,” IEEE Trans. Image Proc., Vol. 6, pp. 298–311, 1997.Google Scholar
  11. 11.
    I. Cohen, “Nonlinear variational method for optical flowcomputation,” in Proc. Eighth Scandinavian Conf. on Image Analysis (SCIA '93, Tromsø, May 25–28, 1993), Vol. 1, 1993, pp. 523–530.Google Scholar
  12. 12.
    N. Cornelius and T. Kanade, “Adapting optical flow to measure object motion in reflectance and X-ray image sequences,” in Proc. ACM SIGRAPH/SIGART Interdisciplinary Workshop on Motion: Representation and Perception (Toronto, April 1983), 1983, pp. 50–58.Google Scholar
  13. 13.
    G. Demoment, “Image reconstruction and restoration: Overview of common estimation structures and problems,” IEEE Trans. Acoustics, Speech, and Signal Processing, Vol. 37, pp. 2024–2036, 1989.Google Scholar
  14. 14.
    R. Deriche, P. Kornprobst, and G. Aubert, “Optical-flow estimation while preserving its discontinuities: A variational approach,” in Proc. Second Asian Conf. Computer Vision, (ACCV'95, Singapore, December 5–8, 1995), Vol. 2, 1995, pp. 290–295.Google Scholar
  15. 15.
    D.J. Fleet and A.D. Jepson, “Computation of component image velocity from local phase information,” Int. J. Comput. Vision, Vol. 5, pp. 77–104, 1990.Google Scholar
  16. 16.
    B. Galvin, B. McCane, K. Novins, D. Mason, and S. Mills, “Recovering motion fields: An analysis of eight optical flow algorithms,” in Proc. 1998 British Machine Vision Conference (BMVC '98, Southampton, September 14–17, 1998).Google Scholar
  17. 17.
    D. Geman and G. Reynolds, “Constrained restoration and the recovery of discontinuities,” IEEE Trans. Pattern Anal. Mach. Intell., Vol. 14, pp. 367–383, 1992.Google Scholar
  18. 18.
    G. Gerig, O. Kübler, R. Kikinis, and F.A. Jolesz, “Nonlinear anisotropic filtering of MRI data,” IEEE Trans. Medical Imaging, Vol. 11, pp. 221–232, 1992.Google Scholar
  19. 19.
    B.M. ter Haar Romeny (Ed.), Geometry-Driven Diffusion in Computer Vision, Kluwer: Dordrecht, 1994.Google Scholar
  20. 20.
    B. Horn and B. Schunck, “Determining optical flow,” Artif. Intell., Vol. 17, pp. 185–203, 1981.Google Scholar
  21. 21.
    A. Kumar, A.R. Tannenbaum, and G.J. Balas, “Optic flow: A curve evolution approach,” IEEE Trans. Image Proc., Vol. 5, pp. 598–610, 1996.Google Scholar
  22. 22.
    P.J.M. van Laarhoven and E.H.L. Aarts, Simulated Annealing: Theory and Applications, Reidel: Dordrecht, 1988.Google Scholar
  23. 23.
    B. Lucas and T. Kanade, “An iterative image registration technique with an application to stereo vision,” in Proc. Seventh Int. Joint Conf. on Artificial Intelligence (IJCAI '81, Vancouver, August 1981), 1981, pp. 674–679.Google Scholar
  24. 24.
    A. Mitiche and P. Bouthemy, “Computation and analysis of image motion: A synopsis of current problems and methods,” Int. J. Comput. Vision, Vol. 19, pp. 29–55, 1996.Google Scholar
  25. 25.
    C.R. Moloney and E. Dubois, “Estimation of motion fields from image sequences with illumination variation,” in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP-93, Toronto, May 1993), Vol. 4, 1993, pp. 2425–2428.Google Scholar
  26. 26.
    K.W. Morton and D.F. Mayers, Numerical Solution of Partial Differential Equations, Cambridge University Press: Cambridge, 1994.Google Scholar
  27. 27.
    D.W. Murray and B.F. Buxton, “Scene segmentation from visual motion using global optimization,” IEEE Trans. Pattern Anal. Mach. Intell., Vol. 9, pp. 220–228, 1987.Google Scholar
  28. 28.
    H.H. Nagel, “Constraints for the estimation of displacement vector fields from image sequences,” in Proc. Eighth Int. Joint Conf. on Artificial Intelligence (IJCAI '83, Karlsruhe, August 8–12, 1983), 1983, pp. 945–951.Google Scholar
  29. 29.
    H.H. Nagel, “Extending the ‘oriented smoothness constraint’ into the temporal domain and the estimation of derivatives of optical flow,” in Computer Vision—ECCV '90, Lecture Notes in Computer Science, Vol. 427, O. Faugeras (Ed.), Springer: Berlin 1990, pp. 139–148.Google Scholar
  30. 30.
    S. Negahdaripour and C.M. Yu, “A generalized brightness change model for computing optical flow,” in Proc. Fourth International Conference on Computer Vision (ICCV '93, Berlin, May 1993), 1993, pp. 2–11.Google Scholar
  31. 31.
    P. Perona and J. Malik, “Scale space and edge detection using anisotropic diffusion,” IEEE Trans. Pattern Anal. Mach. Intell., Vol. 12, pp. 629–639, 1990.Google Scholar
  32. 32.
    M. Proesmans, L. Van Gool, E. Pauwels, and A. Oosterlinck, “Determination of optical flow and its discontinuities using non-linear diffusion,” Computer Vision—ECCV '94, Volume II, Lecture Notes in Computer Science, Vol. 801, J.-O. Eklundh (Ed.), Springer: Berlin, pp. 295–304, 1994.Google Scholar
  33. 33.
    L.I. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noise removal algorithms,” Physica D, Vol. 60, pp. 259–268, 1992.Google Scholar
  34. 34.
    C. Schnörr, “Determining optical flow for irregular domains by minimizing quadratic functionals of a certain class,” Int. J. Comput. Vision, Vol. 6, pp. 25–38, 1991.Google Scholar
  35. 35.
    C. Schnörr, “Unique reconstruction of piecewise smooth images by minimizing strictly convex nonquadratic functionals,” J. Math. Imag. Vision, Vol. 4, pp. 189–198, 1994.Google Scholar
  36. 36.
    C. Schnörr, “Segmentation of visual motion by minimizing convex non-quadratic functionals,” in Proc. 12th Int. Conf. Pattern Recognition (ICPR 12, Jerusalem, Oct. 9–13, 1994), Vol. A, IEEE Computer Society Press: Los Alamitos, 1994, pp. 661–663.Google Scholar
  37. 37.
    M.A. Snyder, “On the mathematical foundations of smoothness constraints for the determination of optical flow and for surface reconstruction,” IEEE Trans.Pattern Anal.Mach. Intell., Vol. 13, pp. 1105–1114, 1991.Google Scholar
  38. 38.
    R.L. Stevenson, B.E. Schmitz, and E.J. Delp, “Discontinuity preserving regularization of inverse visual problems,” IEEE Trans. Systems, Man and Cybernetics, Vol. 24, pp. 455–469, 1994.Google Scholar
  39. 39.
    C. Stiller and J. Konrad, “Estimating motion in image sequences,” IEEE Signal Proc. Magazine, Vol. 16, pp. 70–91, 1999.Google Scholar
  40. 40.
    A. Thompson and D.M. Titterington, “A study of choosing the smoothing parameters in image reconstruction by regularization,” IEEE Trans. Pattern Anal. Mach. Intell., Vol. 13, pp. 326–339, 1991.Google Scholar
  41. 41.
    S. Uras, F. Girosi, A. Verri, and V. Torre, “A computational approach to motion perception,” Biol. Cybern., Vol. 60, pp. 79–87, 1988.Google Scholar
  42. 42.
    J. Weber and J. Malik, “Robust computation of optical flow in a multi-scale differential framework,” Int. J. Comput. Vision, Vol. 14, pp. 67–81, 1995.Google Scholar
  43. 43.
    J.Weickert, Anisotropic Diffusion in Image Processing, Teubner-Verlag: Stuttgart, 1998.Google Scholar
  44. 44.
    J. Weickert, “On discontinuity-preserving optic flow,” in Proc. Computer Vision and Mobile Robotics Workshop (CVMR '98, Santorini, Sept. 17–18, 1998), S. Orphanoudakis, P. Trahanias, J. Crowley, and N. Katevas (Eds.), 1998, pp. 115–122.Google Scholar
  45. 45.
    J. Weickert, B.M. ter Haar Romeny, and M.A. Viergever, “Efficient and reliable schemes for nonlinear diffusion filtering,” IEEE Trans. Image Proc., Vol. 7, pp. 398–410, 1998.Google Scholar
  46. 46.
    J. Weickert, J. Heers, C. Schnörr, K.J. Zuiderveld, O. Scherzer, and H.S. Stiehl, “Fast parallel algorithms for a broad class of nonlinear variational diffusion approaches,” in Real-Time Imaging, Vol. 7, pp. 31–45, 2001.Google Scholar
  47. 47.
    J. Weickert and C. Schnörr, Räumlich-zeitliche Berechnung des optischen Flusses mit nichtlinearen flußtiabhängigen Glattheitstermen, Mustererkennung 1999, W. Förstner, J.M. Buhmann, A. Faber, and P. Faber (Eds.), Springer: Berlin, 1999, pp. 317–324.Google Scholar
  48. 48.
    J. Weickert and C. Schnörr, “A theoretical framework for convex regularizers in PDE-based computation of image motion,” Int. J. Comput. Vision. in press.Google Scholar

Copyright information

© Kluwer Academic Publishers 2001

Authors and Affiliations

  • Joachim Weickert
    • 1
  • Christoph Schnörr
    • 1
  1. 1.Computer Vision, Graphics, and Pattern Recognition Group, Department of Mathematics and Computer ScienceUniversity of MannheimMannheimGermany

Personalised recommendations