International Journal of Computer Vision

, Volume 61, Issue 3, pp 211–231 | Cite as

Lucas/Kanade Meets Horn/Schunck: Combining Local and Global Optic Flow Methods

  • Andrés Bruhn
  • Joachim Weickert
  • Christoph Schnörr

Abstract

Differential methods belong to the most widely used techniques for optic flow computation in image sequences. They can be classified into local methods such as the Lucas–Kanade technique or Bigün's structure tensor method, and into global methods such as the Horn/Schunck approach and its extensions. Often local methods are more robust under noise, while global techniques yield dense flow fields. The goal of this paper is to contribute to a better understanding and the design of novel differential methods in four ways; (i) We juxtapose the role of smoothing/regularisation processes that are required in local and global differential methods for optic flow computation. (ii) This discussion motivates us to describe and evaluate a novel method that combines important advantages of local and global approaches: It yields dense flow fields that are robust against noise. (iii) Spatiotemporal and nonlinear extensions as well as multiresolution frameworks are presented for this hybrid method. (iv) We propose a simple confidence measure for optic flow methods that minimise energy functionals. It allows to sparsify a dense flow field gradually, depending on the reliability required for the resulting flow. Comparisons with experiments from the literature demonstrate the favourable performance of the proposed methods and the confidence measure.

optic flow differential techniques variational methods structure tensor partial differential equations confidence measures performance evaluation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Alvarez, L., Esclarín, J., Lefébure, M., and Sánchez, J. 1999. A PDE model for computing the optical flow. In Proc. XVI Congreso de Ecuaciones Diferenciales y Aplicaciones, Las Palmas de Gran Canaria, Spain. pp. 1349–1356.Google Scholar
  2. Alvarez, L., Weickert, J., and Sánchez., J. 2000. Reliable estimation of dense optical flowfields with large displacements. International Journal of Computer Vision, 39(1):41–56.Google Scholar
  3. Ananden., P. 1989. A computational framework and an algorithm for the measurement of visual motion. International Journal of Computer Vision, 2:283–310.Google Scholar
  4. Aubert, G., Deriche, R., and Kornprobst., P. 1999. Computing optical flow via variational techniques. SIAM Journal on Applied Mathematics, 60(1):156–182.Google Scholar
  5. Bab-Hadiashar, A. and Suter, D. 1998. Robust optic flow computation. International Journal of Computer Vision, 29(1):59–77.Google Scholar
  6. Bainbridge-Smith, A. and Lane, R.G. 1997. Determining optical flow using a differential method. Image and Vision Computing, 15(1):11–22.Google Scholar
  7. Barron, J.L., Fleet, D.J., and Beauchemin, S.S. 1994. Performance of optical flowtechniques. International Journal of Computer Vision, 12(1):43–77.Google Scholar
  8. Bertero, M., Poggio, T.A., and Torre, V. 1988. Ill-posed problems in early vision. Proceedings of the IEEE, 76(8):869–889.Google Scholar
  9. Bigün, J. and Granlund, G.H. 1988. Optical flow based on the inertia matrix in the frequency domain. In Proc. SSAB Symposium on Picture Processing, Lund, Sweden.Google Scholar
  10. Bigün, J., Granlund, G.H., and Wiklund, J. 1991. Multidimensional orientation estimation with applications to texture analysis and optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(8):775–790.Google Scholar
  11. Black, M.J. 1994. Recursive non-linear estimation of discontinuous flow fields. In Computer Vision, ECCV'94, J.-O. Eklundh (Ed.), vol. 800 of Lecture Notes in Computer Science, Springer: Berlin, pp. 138–145.Google Scholar
  12. Black, M.J. and Anandan, P. 1991. Robust dynamic motion estimation over time. In Proc. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society Press: Maui, HI, pp. 292–302.Google Scholar
  13. Black, M.J. and Anandan, P. 1996. The robust estimation of multiple motions: Parametric and piecewise smooth flow fields. Computer Vision and Image Understanding, 63(1):75–104.CrossRefGoogle Scholar
  14. Black, M.J. and Jepson, A. 1996. Estimating optical flow in segmented images using variable-order parametric models with local deformations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(10):972–986.Google Scholar
  15. Brox, T. and Weickert, J. 2002. Nonlinear matrix diffusion for optic flow estimation. In Pattern Recognition, L. Van Gool, (Ed.), vol. 2449 of Lecture Notes in Computer Science, Springer: Berlin, pp. 446–453.Google Scholar
  16. Bruhn, A. 2001. Regularization in motion estimation. Master's thesis, Department of Mathematics and Computer Science, University of Mannheim, Germany.Google Scholar
  17. Bruhn, A., Weickert, J., Feddern, C., Kohlberger, T., and Schnörr, C. 2003. Real-time optic flow computation with variational methods. In Computer Analysis of Images and Patterns, N. Petkov and M.A. Westberg (Eds.), vol. 2756 of Lecture Notes in Computer Science, Springer: Berlin, pp. 222–229.Google Scholar
  18. Bruhn, A., Weickert, J., and SchnÖrr, C. 2002. Combining the advantages of local and global optic flow methods. In Pattern Recognition, L. Van Gool, (Ed.), vol. 2449 of Lecture Notes in Computer Science, Springer: Berlin, pp. 454–462.Google Scholar
  19. Charbonnier, P., Blanc-Féraud, L. Aubert, G., and Barlaud, M. 1994. Two deterministic half-quadratic regularization algorithms for computed imaging. In Proc. 1994 IEEE International Conference on Image Processing, vol. 2. IEEE Computer Society Press: Austin TX, pp. 168–172.Google Scholar
  20. Cohen, I. 1993. Nonlinear variational method for optical flow computation. In Proc. Eighth Scandinavian Conference on Image Analysis, vol. 1, Tromsø, Norway, pp. 523–530.Google Scholar
  21. Courant, R. and Hilbert, D. 1953. Methods of Mathematical Physics, vol. 1. Interscience: New York.Google Scholar
  22. Elad, M. and Feuer, A. 1998. Recursive optical flow estimation—adaptive filtering approach. Journal of Visual Communication and Image Representation, 9(2):119–138.Google Scholar
  23. Elsgolc, L.E. 1961. Calculus of Variations. Pergamon: Oxford.Google Scholar
  24. Farnebäck, G. 2000. Fast and accurate motion estimation using orientation tensors and parametric motion models. In Proc. 15th International Conference on Pattern Recognition, vol. 1, Barcelona, Spain, pp, 135–139.Google Scholar
  25. Farnebäck, G. 2001. Very high accuracy velocity estimation using orientation tensors, parametric motion, and simultaneous segmentation of the motion field. In Proc. Eighth International Conference on Computer Vision, vol. 1, IEEE Computer Society Press: Vancouver, Canada, pp. 171–177.Google Scholar
  26. Fermüller, C., Shulman, D., and Aloimonos, Y. 2001. The statistics of optical flow. Computer Vision and Image Understanding, 82(1):1–32.Google Scholar
  27. Fleet, D.J. and Jepson, A.D. 1990. Computation of component image velocity from local phase information. International Journal of Computer Vision, 5(1):77–104.Google Scholar
  28. Galvin, B., McCane, B., Novins, K., Mason, D., and Mills, S. 1998. Recovering motion fields: An analysis of eight optical flow algorithms. In Proc. 1998 British Machine Vision Conference, Southampton, England.Google Scholar
  29. Hampel, F.R., Ronchetti, E.M., Rousseeuw, P.J., and Stahel, W.A. 1986. Robust Statistics: The Approach Based on Influence Functions. MIT Press: Cambridge, MA.Google Scholar
  30. Heitz, F. and Bouthemy, P. 1993. Multimodal estimation of discontinuous optical flow using Markov random fields. IEEE Transac tions on Pattern Analysis and Machine Intelligence, 15(12):1217–1232.Google Scholar
  31. Hinterberger, W., Scherzer, O., SchnÖrr, C., and Weickert, J. 2002. Analysis of optical flow models in the framework of calculus of variations. Numerical Functional Analysis and Optimization, 23(1/2):69–89.Google Scholar
  32. Horn, B. and Schunck, B. 1981. Determining optical flow. Artificial Intelligence, 17:185–203.CrossRefGoogle Scholar
  33. Huber, P.J. 1981. Robust Statistics. Wiley: New York.Google Scholar
  34. Jähne, B. 2001. Digitale Bildverarbeitung. Springer: Berlin.Google Scholar
  35. Ju, S., Black, M., and Jepson, A. 1996. Skin and bones: Multi-layer, locally affine, optical flow and regularization with transparency. In Proc. 1996 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society Press: San Francisco, CA, pp. 307–314.Google Scholar
  36. Karlholm, J. 1998. local signal models for image sequence analysis. PhD thesis, LinkÖping University, Sweden, Dissertation No. 536.Google Scholar
  37. Kearney, J.K., Thompson, W.B., and Boley, D.L. 1987. Optical flow estimation: An error analysis of gradient-based methods with local optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9(2):229–244.Google Scholar
  38. Kumar, A., Tannenbaum, A.R., and Balas, G.J. 1996. Optic flow: A curve evolution approach. IEEE Transactions on Image Processing, 5(4):598–610.Google Scholar
  39. Lai, S.-H. and Vemuri, B.C. 1998. Reliable and efficient computation of optical flow. International Journal of Computer Vision, 29(2):87–105.Google Scholar
  40. Lauze, F.-B, Kornprobst, P., Lenglet, C., Deriche, R., and Nielsen, M. 2004. Sur quelques méthodes de calcul de flot optique à partir du tenseur de structure: Synthése et contribution. In Proc. 14th French Conference on Pattern Recognition and Artificial Intelligence, Toulouse, France.Google Scholar
  41. Lucas, B. and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In Proc. Seventh International Joint Conference on Artificial Intelligence, Vancouver, Canada, pp. 674–679.Google Scholar
  42. Lucas, B.D. 1984. Generalized image matching by the method of differences. PhD thesis, School of Computer Science, Carnegie-Mellon University, Pittsburgh, PA.Google Scholar
  43. Mémin, E. and Pérez, P. 1998. Dense estimation and object-based segmentation of the optical flow with robust techniques. IEEE Transactions on Image Processing, 7(5):703–719.Google Scholar
  44. Mémin, E. and Pérez, P. 1998. A multigrid approach for hierarchical motion estimation. In Proc. 6th International Conference on Computer Vision, Bombay, India, pp. 933–938.Google Scholar
  45. Mémin, E. and Pérez, P. 2002. Hierarchical estimation and segmentation of dense motion fields. International Journal of Computer Vision, 46(2):129–155.Google Scholar
  46. Mitiche, A. and Bouthemy, P. 1996. Computation and analysis of image motion: A synopsis of current problems and methods. International Journal of Computer Vision, 19(1):29–55.Google Scholar
  47. Murray, D.W. and Buxton, B.F. 1987. Scene segmentation from visual motion using global optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9(2):220–228.Google Scholar
  48. Nagel, H.-H. 1983. Constraints for the estimation of displacement vector fields from image sequences. In Proc. Eighth International Joint Conference on Artificial Intelligence, vol. 2, Karlsruhe, West Germany, pp. 945–951.Google Scholar
  49. Nagel, H.-H. 1990. Extending the ‘oriented smoothness constraint’ into the temporal domain and the estimation of derivatives of optical flow. In Computer Vision-ECCV '90, vol. 427 of Lecture Notes in Computer Science, O. Faugeras (Ed.), Springer: Berlin, pp. 139–148.Google Scholar
  50. Nagel, H.-H. and Gehrke, A. 1998. Spatiotemporally adaptive estimation and segmentation of OF-fields. In Computer Vision-ECCV '98, vol. 1407 of Lecture Notes in Computer Science, H. Burkhardt and B. Neumann, (Eds.), Springer: Berlin, pp. 86–102.Google Scholar
  51. Nesi, P. 1993. Variational approach to optical flowestimation managing discontinuities. Image and Vision Computing, 11(7):419–439.Google Scholar
  52. Ohta, N. 1996. Uncertainty models of the gradient constraint for optical flow computation. IEICE Transactions on Information and Systems, E79-D(7):958–962.Google Scholar
  53. Ong, E.P. and Spann, M. 1999. Robust optical flow computation based on least-median-of-squares regression. International Journal of Computer Vision, 31(1):51–82.Google Scholar
  54. Press, W.H., Teukolsky, S.A., Vetterling, W.T., and Flannery, B.P. 1992. Numerical Recipes in C, 2nd ed. Cambridge University Press: Cambridge, UK.Google Scholar
  55. Proesmans, M., Van Gool, L., Pauwels, E., and Oosterlinck, A. 1994. Determination of optical flow and its discontinuities using non-linear diffusion. In J.-O. Eklundh, (Ed.), Computer Vision-ECCV '94, volume 801 of Lecture Notes in Computer Science. Springer: Berlin, pp. 295–304.Google Scholar
  56. Schnörr, C. 1991. Determining optical flow for irregular domains by minimizing quadratic functionals of a certain class. International Journal of Computer Vision, 6(1):25–38.Google Scholar
  57. Schnörr, C. 1993. On functionals with greyvalue-controlled smoothness terms for determining optical flow. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15:1074–1079.Google Scholar
  58. Schnörr, C. 1994. Segmentation of visual motion by minimizing convex non-quadratic functionals. In Proc. Twelfth International Conference on Pattern Recognition, vol. A, IEEE Computer Society Press: Jerusalem, Israel, pp. 661–663.Google Scholar
  59. Shulman, D. and Hervé, J. 1989. Regularization of discontinuous flow fields. In Proc. Workshop on Visual Motion, IEEE Computer Society Press: Irvine, CA, pp. 81–90.Google Scholar
  60. Simoncelli, E.P., Adelson, E.H., and Heeger, D.J. 1991. Probability distributions of optical flow. In Proc. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society Press: Maui, HI, pp. 310–315.Google Scholar
  61. Stiller, C. and Konrad, J. 1999. Estimating motion in image sequences. IEEE Signal Processing Magazine, 16:70–91.Google Scholar
  62. Szeliski, R. and Coughlan, J. 1994. Hierarchical spline-based image registration. In Proc. 1994 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE Computer Society Press: Seattle, WA, pp. 194–201.Google Scholar
  63. Tretiak, O. and Pastor, L. 1984. Velocity estimation from image sequences with second order differential operators. In Proc. Seventh International Conference on Pattern Recognition, Montreal, Canada, pp. 16–19.Google Scholar
  64. Uras, S., Girosi, F., Verri, A., and Torre, V.A., 1988. computational approach to motion perception. Biological Cybernetics, 60:79–87.Google Scholar
  65. Weber, J. and Malik, J. 1995. Robust computation of optical flow in a multi-scale differential framework. International Journal of Computer Vision, 14:67–81.Google Scholar
  66. Weickert, J. 1998. Anisotropic Diffusion in Image Processing. Teubner, Stuttgart.Google Scholar
  67. Weickert, J. and SchnÖrr, C. 2001. A theoretical framework for convex regularizers in PDE-based computation of image motion. International Journal of Computer Vision, 45(3):245–264.Google Scholar
  68. Weickert, J. and SchnÖrr, C. 2001. Variational optic flow computation with a spatio-temporal smoothness constraint. Journal of Mathematical Imaging and Vision, 14(3):245–255.Google Scholar
  69. Yacoob, Y. and Davis, L.S. 1999. Temporal multi-scale models for flow and acceleration. International Journal of Computer Vision, 32(2):1–17.Google Scholar
  70. Young, D.M. 1971. Iterative Solution of Large Linear Systems. Academic Press: New York.Google Scholar
  71. Yuille, A.L. and Poggio, T.A. 1986. Scaling theorems for zero crossings. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8(1):15–25.Google Scholar

Copyright information

© Kluwer Academic Publishers 2005

Authors and Affiliations

  • Andrés Bruhn
    • 1
  • Joachim Weickert
    • 1
  • Christoph Schnörr
    • 2
  1. 1.Mathematical Image Analysis Group, Faculty of Mathematics and Computer ScienceSaarland UniversitySaarbrückenGermany
  2. 2.Computer Vision, Graphics and Pattern Recognition Group, Faculty of Mathematics and Computer ScienceUniversity of MannheimMannheimGermany

Personalised recommendations