International Journal of Computer Vision

, Volume 67, Issue 2, pp 141–158

Highly Accurate Optic Flow Computation with Theoretically Justified Warping

  • Nils Papenberg
  • Andrés Bruhn
  • Thomas Brox
  • Stephan Didas
  • Joachim Weickert
Article

Abstract

In this paper, we suggest a variational model for optic flow computation based on non-linearised and higher order constancy assumptions. Besides the common grey value constancy assumption, also gradient constancy, as well as the constancy of the Hessian and the Laplacian are proposed. Since the model strictly refrains from a linearisation of these assumptions, it is also capable to deal with large displacements. For the minimisation of the rather complex energy functional, we present an efficient numerical scheme employing two nested fixed point iterations. Following a coarse-to-fine strategy it turns out that there is a theoretical foundation of so-called warping techniques hitherto justified only on an experimental basis. Since our algorithm consists of the integration of various concepts, ranging from different constancy assumptions to numerical implementation issues, a detailed account of the effect of each of these concepts is included in the experimental section. The superior performance of the proposed method shows up by significantly smaller estimation errors when compared to previous techniques. Further experiments also confirm excellent robustness under noise and insensitivity to parameter variations.

Keywords

optical flow differential methods gradient constancy warping performance evaluation 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Acar, R. and Vogel, C.R. 1994. Analysis of bounded variation penalty methods for ill-posed problems. Inverse Problems, 10:1217–1229.CrossRefMathSciNetMATHGoogle Scholar
  2. Alvarez, L., Esclarín, J., Lefébure, M., and Sánchez, J. 1999. A PDE model for computing the optical flow. In Proc. XVI Congreso de Ecuaciones Diferenciales y Aplicaciones, Las Palmas de Gran Canaria, Spain, pp. 1349–1356.Google Scholar
  3. Alvarez, L., Weickert, J. and Sánchez, J. 2000. Reliable estimation of dense optical flow fields with large displacements. International Journal of Computer Vision, 39(1):41–56.CrossRefMATHGoogle Scholar
  4. Anandan, P. 1989. A computational framework and an algorithm for the measurement of visual motion. International Journal of Computer Vision, 2:283–310.CrossRefGoogle Scholar
  5. Aubert, G., Deriche, R. and Kornprobst, P. 1999. Computing optical flow via variational techniques. SIAM Journal on Applied Mathematics, 60(1):156–182.CrossRefMathSciNetMATHGoogle Scholar
  6. Bab-Hadiashar, A. and Suter, D. 1998. Robust optic flow computation. International Journal of Computer Vision, 29(1):59–77.CrossRefGoogle Scholar
  7. Barron, J.L., Fleet, D.J., and Beauchemin, S.S. 1994. Performance of optical flow techniques. International Journal of Computer Vision, 12(1):43–77.CrossRefGoogle Scholar
  8. Bertero, M., Poggio, T.A., and Torre, V. 1988. Ill-posed problems in early vision. Proc. of the IEEE, 76(8):869–889.CrossRefGoogle Scholar
  9. Black, M.J. 1994. Recursive non-linear estimation of discontinuous flow fields. In Computer Vision – ECCV ’94, J.-O. Eklundh (Ed.) vol. 800 of Lecture Notes in Computer Science, pp. 138–145. Springer, Berlin.Google Scholar
  10. Black, M.J. and Anandan, P. 1991. Robust dynamic motion estimation over time. In Proc. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Maui, HI, June, IEEE Computer Society Press, pp. 292–302.Google Scholar
  11. Black, M.J. and Anandan, P. 1996. The robust estimation of multiple motions: parametric and piecewise smooth flow fields. Computer Vision and Image Understanding, 63(1):75–104.CrossRefGoogle Scholar
  12. Black, M.J. and Jepson, A. 1996. Estimating optical flow in segmented images using variable-order parametric models with local deformations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(10):972–986.CrossRefGoogle Scholar
  13. Brox, T. et al. 2004. High accuracy optical flow estimation based on a theory for warping. In Computer Vision, Proc. 8th European Conference on Computer Vision, T. Pajdla and J. Matas (Ed.) vol. 3024 of Lecture Notes in Computer Science, Springer, Prague, Czech Republic, pp. 25–36.Google Scholar
  14. Bruhn, A., Weickert, J., and Schnörr, C. 2005. Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. International Journal of Computer Vision, 61(3):211–231.CrossRefGoogle Scholar
  15. Cohen, I. 1993. Nonlinear variational method for optical flow computation. In Proc. Eighth Scandinavian Conference on Image Analysis, Tromsø, Norway, vol. 1, pp. 523–530.Google Scholar
  16. Cremers, D. 2003. A multiphase levelset framework for variational motion segmentation. In Scale Space Methods in Computer Vision, L. D. Griffin and M. Lillholm (Ed.) vol. 2695 of Lecture Notes in Computer Science, Springer, Berlin, pp. 599–614.Google Scholar
  17. Deriche, R., Kornprobst, P., and Aubert, G. 1995. Optical-flow estimation while preserving its discontinuities: a variational approach. In Proc. Second Asian Conference on Computer Vision, Singapore, vol. 2, pp. 290–295.Google Scholar
  18. Elad, M. and Feuer, A. 1998. Recursive optical flow estimation – adaptive filtering approach. Journal of Visual Communication and Image Representation, 9(2):119–138.CrossRefGoogle Scholar
  19. Farnebäck, G. 2000. Fast and accurate motion estimation using orientation tensors and parametric motion models. In Proc. 15th International Conference on Pattern Recognition, Barcelona, Spain, vol. 1, pp. 135–139.Google Scholar
  20. Farnebäck, G. 2001. Very high accuracy velocity estimation using orientation tensors, parametric motion, and simultaneous segmentation of the motion field. In Proc. Eighth International Conference on Computer Vision, Vancouver, Canada, July IEEE Computer Society Press, vol. 1, pp. 171–177.Google Scholar
  21. Hampel, F.R., et al. 1986. Robust Statistics: The Approach Based on Influence Functions. MIT Press, Cambridge, MA.MATHGoogle Scholar
  22. Heitz, F. and Bouthemy, P. 1993. Multimodal estimation of discontinuous optical flow using Markov random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(12):1217–1232.CrossRefGoogle Scholar
  23. Horn, B. and Schunck, B. 1981. Determining optical flow. Artificial Intelligence, 17:185–203.CrossRefGoogle Scholar
  24. Huber, P.J. 1981. Robust Statistics. Wiley, New York.MATHGoogle Scholar
  25. Ju, S., Black, M., and Jepson, 1996. A. Skin and bones: multi-layer, locally affine, optical flow and regularization with transparency. In Proc. 1996 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, June IEEE Computer Society Press, pp. 307–314.Google Scholar
  26. Kumar, A., Tannenbaum, A.R., and Balas, G.J. 1996. Optic flow: a curve evolution approach. IEEE Transactions on Image Processing, 5(4):598–610.CrossRefGoogle Scholar
  27. Lai, S.-H. and Vemuri, B.C. 1998. Reliable and efficient computation of optical flow. International Journal of Computer Vision, 29(2):87–105.CrossRefGoogle Scholar
  28. Lefébure, M. and Cohen, L.D. 2001. Image registration, optical flow and local rigidity. Journal of Mathematical Imaging and Vision, 14(2):131–147.CrossRefMATHGoogle Scholar
  29. Lucas, B. and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In Proc. Seventh International Joint Conference on Artificial Intelligence, Vancouver, Canada, pp. 674–679.Google Scholar
  30. Mémin, E. and Pérez, P. 1998. Dense estimation and object-based segmentation of the optical flow with robust techniques. IEEE Transactions on Image Processing, 7(5):703–719.CrossRefGoogle Scholar
  31. Mémin, E. and Pérez, P. 1998. A multigrid approach for hierarchical motion estimation. In Proc. Sixth International Conference on Computer Vision, Bombay, India, Jan. Narosa Publishing House, pp. 933–938.Google Scholar
  32. Mémin, E. and Pérez, P. 2002. Hierarchical estimation and segmentation of dense motion fields. International Journal of Computer Vision, 46(2):129–155.CrossRefMATHGoogle Scholar
  33. Murray, D.W. and Buxton, B.F. 1987. Scene segmentation from visual motion using global optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9(2):220–228.CrossRefGoogle Scholar
  34. Nagel, H.-H. 1983. Constraints for the estimation of displacement vector fields from image sequences. In Proc. Eighth International Joint Conference on Artificial Intelligence, Karlsruhe, West Germany, vol. 2, pp. 945–951.Google Scholar
  35. Nagel, H.-H. 1990. Extending the ‘oriented smoothness constraint’ into the temporal domain and the estimation of derivatives of optical flow. In Computer Vision - ECCV ’90, O. Faugeras (Ed.) vol. 427 of Lecture Notes in Computer Science, Springer, Berlin, pp. 139–148.Google Scholar
  36. Nagel, H.-H. and Enkelmann, W. 1986. An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8:565–593.Google Scholar
  37. Nesi, P. 1993. Variational approach to optical flow estimation managing discontinuities. Image and Vision Computing, 11(7):419–439.CrossRefGoogle Scholar
  38. Proesmans, M., et al. 1994. Determination of optical flow and its discontinuities using non-linear diffusion. In Computer Vision – ECCV ’94, J.-O. Eklundh (Ed.) vol. 801 of Lecture Notes in Computer Science, Springer, Berlin, pp. 295–304.Google Scholar
  39. Rudin, L.I., Osher, S., and Fatemi, E. 1992. Nonlinear total variation based noise removal algorithms. Physica D, 60:259–268.CrossRefMATHGoogle Scholar
  40. Schnörr, C. 1994. Bewegungssegmentation von Bildfolgen durch die Minimierung konvexer nicht-quadratischer Funktionale. In Mustererkennung 1994, W. Kropatsch and H. Bischof (Ed.) Springer, Berlin, pp. 178–185.Google Scholar
  41. Schnörr, C., 1994. Segmentation of visual motion by minimizing convex non-quadratic functionals. In Proc. Twelfth International Conference on Pattern Recognition, Jerusalem, Israel, Oct. IEEE Computer Society Press, vol. A, pp. 661–663.Google Scholar
  42. Shulman, D. and Hervé, J. Regularization of discontinuous flow fields. In Proc. Workshop on Visual Motion, Irvine, CA, IEEE Computer Society Press, pp. 81–86.Google Scholar
  43. Szeliski, R. and Coughlan, J. 1994. Hierarchical spline-based image registration. In Proc. 1994 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Seattle, WA, June IEEE Computer Society Press, pp. 194–201.Google Scholar
  44. Tistarelli, M. 1994. Multiple constraints for optical flow. In Computer Vision – ECCV ’94, J.-O. Eklundh (Ed.) vol. 800 of Lecture Notes in Computer Science, Springer, Berlin, pp. 61–70.Google Scholar
  45. Uras, S., et al. 1988. A computational approach to motion perception. Biological Cybernetics, 60:79–87.CrossRefGoogle Scholar
  46. Weickert, J. and Schnörr, C. 2001. A theoretical framework for convex regularizers in PDE-based computation of imagemotion. International Journal of Computer Vision, 45(3):245–264.CrossRefMATHGoogle Scholar
  47. Weickert, J. and Schnörr, C. 2001. Variational optic flow computation with a spatio-temporal smoothness constraint. Journal of Mathematical Imaging and Vision, 14(3):245–255.CrossRefMATHGoogle Scholar
  48. Young, D. M. Iterative Solution of Large Linear Systems. Academic Press, New York, 1971.MATHGoogle Scholar

Copyright information

© Springer Science + Business Media, Inc. 2006

Authors and Affiliations

  • Nils Papenberg
    • 1
  • Andrés Bruhn
    • 2
  • Thomas Brox
    • 2
  • Stephan Didas
    • 2
  • Joachim Weickert
    • 2
  1. 1.Institute for MathematicsUniversity of LübeckLübeckGermany
  2. 2.Mathematical Image Analysis Group, Faculty of Mathematics and Computer ScienceSaarland UniversitySaarbrückenGermany

Personalised recommendations