International Journal of Computer Vision

, Volume 39, Issue 1, pp 41–56 | Cite as

Reliable Estimation of Dense Optical Flow Fields with Large Displacements

  • Luis Alvarez
  • Joachim Weickert
  • Javier Sánchez

Abstract

In this paper we show that a classic optical flow technique by Nagel and Enkelmann (1986, IEEE Trans. Pattern Anal. Mach. Intell., Vol. 8, pp. 565–593) can be regarded as an early anisotropic diffusion method with a diffusion tensor. We introduce three improvements into the model formulation that (i) avoid inconsistencies caused by centering the brightness term and the smoothness term in different images, (ii) use a linear scale-space focusing strategy from coarse to fine scales for avoiding convergence to physically irrelevant local minima, and (iii) create an energy functional that is invariant under linear brightness changes. Applying a gradient descent method to the resulting energy functional leads to a system of diffusion–reaction equations. We prove that this system has a unique solution under realistic assumptions on the initial data, and we present an efficient linear implicit numerical scheme in detail. Our method creates flow fields with 100 % density over the entire image domain, it is robust under a large range of parameter variations, and it can recover displacement fields that are far beyond the typical one-pixel limits which are characteristic for many differential methods for determining optical flow. We show that it performs better than the optical flow methods with 100 % density that are evaluated by Barron et al. (1994, Int. J. Comput. Vision, Vol. 12, pp. 43–47). Our software is available from the Internet.

image sequences optical flow differential methods anisotropic diffusion linear scale-space regularization finite difference methods performance evaluation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Alvarez, L., Deriche, R., Sánchez, J., and Weickert, J. 2000. Dense disparity map estimation respecting image derivatives: A PDE and scale-space based approach. Report No. 3874, ROBOTVIS, INRIA, 2004 route des Lucioles, BP 93, 06902 Sophia-Antipolis Cedex, France.Google Scholar
  2. Alvarez, L., Esclarin, J., Lefébure, M., and Sá0nchez, J. 1999. A PDE model for computing the optical flow. In Proc. XVI Congreso de Ecuaciones Diferenciales y Aplicaciones, C.E.D.Y.A. XVI, Las Palmas de Gran Canaria, September 21-24, 1999, pp. 1349–1356.Google Scholar
  3. Alvarez, L., Guichard, F., Lions, P.-L., and Morel, J.-M. 1993. Axioms and fundamental equations in image processing. Arch. Rational Mech. Anal., 123:199–257.Google Scholar
  4. Alvarez, L., Lions, P.-L., and Morel, J.-M. 1992. Image selective smoothing and edge detection by nonlinear diffusion II. SIAM J. Numer. Anal., 29:845–866.Google Scholar
  5. Alvarez, L. and Sánchez, J. 2000. 3-D geometry reconstruction using a color image stereo pair and partial differential equations. Report No. 6, Instituto Universitario de Ciencias y Tecnologias Cibernéticas, University of Las Palmas de Gran Canaria, Campus de Tafira, 35017 Las Palmas, Spain.Google Scholar
  6. Alvarez, L., Weickert, J., and Sánchez, J. 1999. A scale-space approach to nonlocal optical flow calculations. In Lecture Notes in Computer Science, Vol. 1682: Scale-Space Theories in Computer Vision, M. Nielsen, P. Johansen, O.F. Olsen, and J. Weickert (Eds.). Springer: Berlin, pp. 235–246.Google Scholar
  7. Anandan, P. 1989. A computational framework and an algorithm for the measurement of visual motion. Int. J. Comput. Vision, 2:283–310.Google Scholar
  8. Aubert, G., Deriche, R., and Kornprobst, P. to appear. Computing optical flow via variational techniques, in SIAM J. Math. Anal.Google Scholar
  9. Barron, J.L., Fleet, D.J., and Beauchemin, S.S. 1994. Performance of optical flow techniques. Int. J. Comput. Vision, 12:43–77.Google Scholar
  10. Berger, M.-O., Deriche, R., Herlin, I., Jaffré, J., and Morel, J.-M. (Eds.). 1996. ICAOS’ 96: Images, wavelets and PDEs. Lecture Notes in Control and Information Sciences, Vol. 219. Springer: London.Google Scholar
  11. Bergholm, F. 1987. Edge focusing, IEEE Trans. Pattern Anal. Mach. Intell., 9:726–741.Google Scholar
  12. Bertero, M., Poggio, T.A., and Torre, V. 1988. Ill-posed problems in early vision. Proc. IEEE, 76:869–889.Google Scholar
  13. Black, M.J. and Anandan, P. 1991. Robust dynamic motion estimation over time. In Proc. IEEE Comp. Soc. Conf. on Computer Vision and Pattern Recognition, CVPR’ 91, Maui, June 3-6, 1991, IEEE Computer Society Press, Los Alamitos, pp. 292–302.Google Scholar
  14. Black, M.J. and Anandan, P. 1996. The robust estimation of multiple motions: Parametric and piecewise smooth flow fields. Computer Vision and Image Understanding, 63:75–104.Google Scholar
  15. Blake, A. and Zisserman, A. 1987. Visual Reconstruction. MIT Press: Cambridge, MA.Google Scholar
  16. Blanc-Féraud, L., Barlaud, M., and Gaidon, T. 1993. Motion estimation involving discontinuities in a multiresolution scheme. Optical Engineering, 32(7):1475–1482.Google Scholar
  17. Brezis, H. 1973. Opérateurs maximaux monotones et semi-groupes de contractions dans les espaces de Hilbert. North Holland: Amsterdam.Google Scholar
  18. Caselles, V., Morel, J.M., Sapiro, G., and Tannenbaum, A. (Eds.). 1998. Special issue on partial differential equations and geometrydriven diffusion in image processing and analysis. IEEE Trans. Image Proc. 7(3).Google Scholar
  19. Cohen, I. 1993. Nonlinear variational method for optical flow computation. In Proc. Eighth Scandinavian Conf. on Image Analysis, SCIA’ 93, Tromsø, May 25-28, 1993, Vol. 1, pp. 523–530.Google Scholar
  20. Deriche, R., Kornprobst, P., and Aubert, G. 1995. Optical-flow estimation while preserving its discontinuities: A variational approach. In Proc. Second Asian Conf. Computer Vision, ACCV’ 95, Singapore, December 5-8, 1995, Vol. 2, pp. 290–295.Google Scholar
  21. Enkelmann, W. 1988. Investigation of multigrid algorithms for the estimation of optical flow fields in image sequences. Computer Vision, Graphics and Image Processing, 43:150–177.Google Scholar
  22. Florack, L. 1997. Image Structure. Kluwer: Dordrecht.Google Scholar
  23. Florack, L.M.J., Niessen, W.J., and Nielsen, M. 1998. The intrinsic structure of the optic flow incorporating measurement duality. Int. J. Comput. Vision, 27:263–286.Google Scholar
  24. Galvin, B., McCane, B., and Novins, K. 1998. Recovering motion fields: An analysis of eight optical flow algorithms. In Proc. 1998 British Machine Vision Conference, BMVC’ 98, Southampton, September 14-17, 1998.Google Scholar
  25. González, E., Trujillo, A., and MegaWave. A window oriented image processing software. Departamento de Informática y Sistemas, Universidad de Las Palmas. http://amiserver.dis.ulpgc.es/xmwgus/, 1994.Google Scholar
  26. Guichard, F. and Rudin, L. 1996. Accurate estimation of discontinuous optical flow by minimizing divergence related functionals. In Proc. Third Int. Conf. on Image Processing, ICIP’ 96, Lausanne, September 1996, Vol. 1, pp. 497–500.Google Scholar
  27. Romeny, B. ter Haar, Florack, L., Koenderink, J., and Viergever, M. (Eds.). 1997. Scale-space theory in computer vision. Lecture Notes in Computer Science, Vol. 1252. Springer: Berlin.Google Scholar
  28. Heitz, F. and Bouthemy, P. 1993. Multimodal estimation of discontinuous optical flow using Markov random fields. IEEE Trans. Pattern Anal. Mach. Intell., 15:1217–1232.Google Scholar
  29. Hinterberger, W. 1999. Generierung eines Films zwischen zwei Bildern mit Hilfe des optischen Flusses. M.Sc. Thesis, Industrial Mathematics Institute, University of Linz, Austria.Google Scholar
  30. Horn, B. and Schunck, B. 1981. Determining optical flow. Artificial Intelligence, 17:185–203.Google Scholar
  31. Iijima, T. 1962. Basic theory on normalization of pattern (in case of typical one-dimensional pattern). Bulletin of the Electrotechnical Laboratory (in Japanese), 26:368–388.Google Scholar
  32. Iijima, T. 1973. Pattern Recognition. (in Japanese) Corona-sha.Google Scholar
  33. Iijima, T. 1989. Series of Basic Information Technology, Vol. 6: Theory of Pattern Recognition (in Japanese). Morishita Publishing.Google Scholar
  34. Jähne, B. and Haussecker, H. 1998. Performance characteristics of low-level motion estimators in spatiotemporal images. In Proc. Workshop on Evaluation and Validation of Computer Vision Algorithms, Dagstuhl, March 16-20, 1998, R. Haralick, R. Klette, H.S. Stiehl, and M. Viergever (Eds.). Kluwer: Dordrecht. in press.Google Scholar
  35. Koenderink, J.J. 1984. The structure of images. Biological Cybernetics, 50:363–370.Google Scholar
  36. Kumar, A., Tannenbaum, A.R., and Balas, G.J. 1996. Optic flow: A curve evolution approach. IEEE Trans. Image Proc., 5:598–610.Google Scholar
  37. Lindeberg. 1994. Scale-Space Theory in ComputerVision. Kluwer: Boston.Google Scholar
  38. Mémin, E. and Pérez, P. 1998. Dense estimation and object-based segmentation of the optical flow with robust techniques. IEEE Trans. Image Proc, 7:703–719.Google Scholar
  39. Mitiche, A. and Bouthemy, P. 1996. Computation and analysis of image motion: A synopsis of current problems and methods. Int. J. Comput. Vision, 19:29–55.Google Scholar
  40. Morton, K.W. and Mayers, L.M. 1994. Numerical Solution of Partial Differential Equations. Cambridge University Press: Cambridge.Google Scholar
  41. Nagel, H.-H. 1983. Constraints for the estimation of displacement vector fields from image sequences. In Proc. Eighth Int. Joint Conf. on Artificial Intelligence, IJCAI’ 83, Karls-ruhe, August 8-12, 1983, pp. 945–951.Google Scholar
  42. Nagel, H.-H. 1987. On the estimation of optical flow: Relations between different approaches and some new results. Artificial Intelligence, 33:299–324.Google Scholar
  43. Nagel, H.-H. 1990. Extending the ‘oriented smoothness constraint’ into the temporal domain and the estimation of derivatives of optical flow. In Lecture Notes in Computer Science, Vol. 427: Computer Vision–ECCV’ 90, O. Faugeras (Ed.). Springer; Berlin, pp. 139–148.Google Scholar
  44. Nagel, H.-H. and Enkelmann, W. 1986. An investigation of smoothness constraints for the estimation of displacement vector fields from images sequences. IEEE Trans. Pattern Anal. Mach. Intell., 8:565–593.Google Scholar
  45. Nesi, P. 1993. Variational approach to optical flow estimation managing discontinuities. Image and Vision Computing, 11:419–439.Google Scholar
  46. Nielsen, M., Johansen, P., Olsen, O.F., and Weickert, J. (Eds.). 1999. Scale-space theories in computer vision. Lecture Notes in Computer Science, Vol. 1682. Springer: Berlin.Google Scholar
  47. Otte, M. and Nagel, H.-H. 1995. Estimation of optical flow based on higher-order spatiotemporal derivatives in interlaced and noninterlaced image sequences. Artificial Intelligence, 78:5–43.Google Scholar
  48. Proesmans, M., Pauwels, E., and Van Gool, L. 1994a. Coupled geometry-driven diffusion equations for low-level vision. In Geometry-Driven Diffusion in Computer Vision, B.M. ter Haar Romeny (Ed.). Kluwer: Dordrecht, pp. 191–228.Google Scholar
  49. Proesmans, M., Van Gool, L., Pauwels, E., and Oosterlinck, A. 1994b. Determination of optical flow and its discontinuities using non-linear diffusion. In Lecture Notes in Computer Science, Vol. 801: Computer Vision–ECCV’ 94, J.-O. Eklundh (Ed.). Springer: Berlin, pp. 295–304.Google Scholar
  50. Schnörr, C. 1991a. Determining optical flowfor irregular domains by minimizing quadratic functionals of a certain class. Int. J. Comput. Vision, 6:25–38.Google Scholar
  51. Schnörr. 1991b.On functionals with greyvalue-controlled smoothness terms for determining optical flow. IEEE Trans. Pattern Anal. Mach. Intell., 15:1074–1079.Google Scholar
  52. Schnörr, C. 1994. Segmentation of visual motion by minimizing convex non-quadratic functionals. In Proc. 12th Int. Conf. Pattern Recognition, ICPR 12, Jerusalem, October 9-13, 1994, EEE Computer Society Press, Los Alamitos, Vol. A, pp. 661–663.Google Scholar
  53. Shah, J. 1993. A nonlinear diffusion model for discontinuous disparity and half-occlusions in stereo. In Proc. IEEE Comp. Soc. Conf. Computer Vision and Pattern Recognition, CVPR’ 93, New York, June 15-17, 1993, IEEE Computer Society Press, Los Alamitos, pp. 34–40.Google Scholar
  54. Snyder, M.A. 1991. On the mathematical foundations of smoothness constraints for the determination of optical flow and for surface reconstruction. IEEE Trans. Pattern Anal. Mach. Intell., 13:1105–1114.Google Scholar
  55. Sporring, J., Nielsen, M., Florack, L., and Johansen, P. (Eds.). 1997. Gaussian Scale-Space Theory. Kluwer: Dordrecht.Google Scholar
  56. Weickert, J. 1996. Theoretical foundations of anisotropic diffusion in image processing. Computing, Suppl. 11:221–236.Google Scholar
  57. Weickert, J. 1998. Anisotropic Diffusion in Image Processing. Teubner; Stuttgart.Google Scholar
  58. Weickert, J. 1998. On discontinuity-preserving optic flow. In Proc. Computer Vision and Mobile RoboticsWorkshop,CVMR'98, Santorini, September 17-18, 1998, S. Orphanoudakis, P. Trahanias, J. Crowley, and N. Katevas (Eds.). pp. 115–122.Google Scholar
  59. Weickert, J., Ishikawa, S., and Imiya, A. 1999. Linear scale-space has first been proposed in Japan. J. Math. Imag. Vision, 10:237–252.Google Scholar
  60. Weickert, J. and Schnörr, C. 1999. Räumlich-zeitliche Berechnung des optischen Flusses mit nichtlinearen flußabhängigen Glattheitstermen. Mustererkennung 1999, W. Förstner, J.M. Buhmann, A. Faber, and P. Faber (Eds.). Springer: Berlin, pp. 317–324.Google Scholar

Copyright information

© Kluwer Academic Publishers 2000

Authors and Affiliations

  • Luis Alvarez
    • 1
  • Joachim Weickert
    • 2
  • Javier Sánchez
    • 1
  1. 1.Departamento de Informática y SistemasUniversidad de Las PalmasLas PalmasSpain
  2. 2.Computer Vision, Graphics, and Pattern Recognition Group, Department of Mathematics and Computer ScienceUniversity of MannheimMannheimGermany

Personalised recommendations