Skip to main content
SpringerLink
Log in

Highly Accurate Optic Flow Computation with Theoretically Justified Warping

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

In this paper, we suggest a variational model for optic flow computation based on non-linearised and higher order constancy assumptions. Besides the common grey value constancy assumption, also gradient constancy, as well as the constancy of the Hessian and the Laplacian are proposed. Since the model strictly refrains from a linearisation of these assumptions, it is also capable to deal with large displacements. For the minimisation of the rather complex energy functional, we present an efficient numerical scheme employing two nested fixed point iterations. Following a coarse-to-fine strategy it turns out that there is a theoretical foundation of so-called warping techniques hitherto justified only on an experimental basis. Since our algorithm consists of the integration of various concepts, ranging from different constancy assumptions to numerical implementation issues, a detailed account of the effect of each of these concepts is included in the experimental section. The superior performance of the proposed method shows up by significantly smaller estimation errors when compared to previous techniques. Further experiments also confirm excellent robustness under noise and insensitivity to parameter variations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Acar, R. and Vogel, C.R. 1994. Analysis of bounded variation penalty methods for ill-posed problems. Inverse Problems, 10:1217–1229.

    Article  MathSciNet  MATH  Google Scholar 

  • Alvarez, L., Esclarín, J., Lefébure, M., and Sánchez, J. 1999. A PDE model for computing the optical flow. In Proc. XVI Congreso de Ecuaciones Diferenciales y Aplicaciones, Las Palmas de Gran Canaria, Spain, pp. 1349–1356.

    Google Scholar 

  • Alvarez, L., Weickert, J. and Sánchez, J. 2000. Reliable estimation of dense optical flow fields with large displacements. International Journal of Computer Vision, 39(1):41–56.

    Article  MATH  Google Scholar 

  • Anandan, P. 1989. A computational framework and an algorithm for the measurement of visual motion. International Journal of Computer Vision, 2:283–310.

    Article  Google Scholar 

  • Aubert, G., Deriche, R. and Kornprobst, P. 1999. Computing optical flow via variational techniques. SIAM Journal on Applied Mathematics, 60(1):156–182.

    Article  MathSciNet  MATH  Google Scholar 

  • Bab-Hadiashar, A. and Suter, D. 1998. Robust optic flow computation. International Journal of Computer Vision, 29(1):59–77.

    Article  Google Scholar 

  • Barron, J.L., Fleet, D.J., and Beauchemin, S.S. 1994. Performance of optical flow techniques. International Journal of Computer Vision, 12(1):43–77.

    Article  Google Scholar 

  • Bertero, M., Poggio, T.A., and Torre, V. 1988. Ill-posed problems in early vision. Proc. of the IEEE, 76(8):869–889.

    Article  Google Scholar 

  • Black, M.J. 1994. Recursive non-linear estimation of discontinuous flow fields. In Computer Vision – ECCV ’94, J.-O. Eklundh (Ed.) vol. 800 of Lecture Notes in Computer Science, pp. 138–145. Springer, Berlin.

    Google Scholar 

  • Black, M.J. and Anandan, P. 1991. Robust dynamic motion estimation over time. In Proc. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Maui, HI, June, IEEE Computer Society Press, pp. 292–302.

    Google Scholar 

  • Black, M.J. and Anandan, P. 1996. The robust estimation of multiple motions: parametric and piecewise smooth flow fields. Computer Vision and Image Understanding, 63(1):75–104.

    Article  Google Scholar 

  • Black, M.J. and Jepson, A. 1996. Estimating optical flow in segmented images using variable-order parametric models with local deformations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(10):972–986.

    Article  Google Scholar 

  • Brox, T. et al. 2004. High accuracy optical flow estimation based on a theory for warping. In Computer Vision, Proc. 8th European Conference on Computer Vision, T. Pajdla and J. Matas (Ed.) vol. 3024 of Lecture Notes in Computer Science, Springer, Prague, Czech Republic, pp. 25–36.

    Google Scholar 

  • Bruhn, A., Weickert, J., and Schnörr, C. 2005. Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. International Journal of Computer Vision, 61(3):211–231.

    Article  Google Scholar 

  • Cohen, I. 1993. Nonlinear variational method for optical flow computation. In Proc. Eighth Scandinavian Conference on Image Analysis, Tromsø, Norway, vol. 1, pp. 523–530.

  • Cremers, D. 2003. A multiphase levelset framework for variational motion segmentation. In Scale Space Methods in Computer Vision, L. D. Griffin and M. Lillholm (Ed.) vol. 2695 of Lecture Notes in Computer Science, Springer, Berlin, pp. 599–614.

    Google Scholar 

  • Deriche, R., Kornprobst, P., and Aubert, G. 1995. Optical-flow estimation while preserving its discontinuities: a variational approach. In Proc. Second Asian Conference on Computer Vision, Singapore, vol. 2, pp. 290–295.

  • Elad, M. and Feuer, A. 1998. Recursive optical flow estimation – adaptive filtering approach. Journal of Visual Communication and Image Representation, 9(2):119–138.

    Article  Google Scholar 

  • Farnebäck, G. 2000. Fast and accurate motion estimation using orientation tensors and parametric motion models. In Proc. 15th International Conference on Pattern Recognition, Barcelona, Spain, vol. 1, pp. 135–139.

  • Farnebäck, G. 2001. Very high accuracy velocity estimation using orientation tensors, parametric motion, and simultaneous segmentation of the motion field. In Proc. Eighth International Conference on Computer Vision, Vancouver, Canada, July IEEE Computer Society Press, vol. 1, pp. 171–177.

    Google Scholar 

  • Hampel, F.R., et al. 1986. Robust Statistics: The Approach Based on Influence Functions. MIT Press, Cambridge, MA.

    MATH  Google Scholar 

  • Heitz, F. and Bouthemy, P. 1993. Multimodal estimation of discontinuous optical flow using Markov random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(12):1217–1232.

    Article  Google Scholar 

  • Horn, B. and Schunck, B. 1981. Determining optical flow. Artificial Intelligence, 17:185–203.

    Article  Google Scholar 

  • Huber, P.J. 1981. Robust Statistics. Wiley, New York.

    MATH  Google Scholar 

  • Ju, S., Black, M., and Jepson, 1996. A. Skin and bones: multi-layer, locally affine, optical flow and regularization with transparency. In Proc. 1996 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, June IEEE Computer Society Press, pp. 307–314.

    Google Scholar 

  • Kumar, A., Tannenbaum, A.R., and Balas, G.J. 1996. Optic flow: a curve evolution approach. IEEE Transactions on Image Processing, 5(4):598–610.

    Article  Google Scholar 

  • Lai, S.-H. and Vemuri, B.C. 1998. Reliable and efficient computation of optical flow. International Journal of Computer Vision, 29(2):87–105.

    Article  Google Scholar 

  • Lefébure, M. and Cohen, L.D. 2001. Image registration, optical flow and local rigidity. Journal of Mathematical Imaging and Vision, 14(2):131–147.

    Article  MATH  Google Scholar 

  • Lucas, B. and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In Proc. Seventh International Joint Conference on Artificial Intelligence, Vancouver, Canada, pp. 674–679.

  • Mémin, E. and Pérez, P. 1998. Dense estimation and object-based segmentation of the optical flow with robust techniques. IEEE Transactions on Image Processing, 7(5):703–719.

    Article  Google Scholar 

  • Mémin, E. and Pérez, P. 1998. A multigrid approach for hierarchical motion estimation. In Proc. Sixth International Conference on Computer Vision, Bombay, India, Jan. Narosa Publishing House, pp. 933–938.

    Google Scholar 

  • Mémin, E. and Pérez, P. 2002. Hierarchical estimation and segmentation of dense motion fields. International Journal of Computer Vision, 46(2):129–155.

    Article  MATH  Google Scholar 

  • Murray, D.W. and Buxton, B.F. 1987. Scene segmentation from visual motion using global optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 9(2):220–228.

    Article  Google Scholar 

  • Nagel, H.-H. 1983. Constraints for the estimation of displacement vector fields from image sequences. In Proc. Eighth International Joint Conference on Artificial Intelligence, Karlsruhe, West Germany, vol. 2, pp. 945–951.

  • Nagel, H.-H. 1990. Extending the ‘oriented smoothness constraint’ into the temporal domain and the estimation of derivatives of optical flow. In Computer Vision - ECCV ’90, O. Faugeras (Ed.) vol. 427 of Lecture Notes in Computer Science, Springer, Berlin, pp. 139–148.

    Google Scholar 

  • Nagel, H.-H. and Enkelmann, W. 1986. An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence, 8:565–593.

    Google Scholar 

  • Nesi, P. 1993. Variational approach to optical flow estimation managing discontinuities. Image and Vision Computing, 11(7):419–439.

    Article  Google Scholar 

  • Proesmans, M., et al. 1994. Determination of optical flow and its discontinuities using non-linear diffusion. In Computer Vision – ECCV ’94, J.-O. Eklundh (Ed.) vol. 801 of Lecture Notes in Computer Science, Springer, Berlin, pp. 295–304.

    Google Scholar 

  • Rudin, L.I., Osher, S., and Fatemi, E. 1992. Nonlinear total variation based noise removal algorithms. Physica D, 60:259–268.

    Article  MATH  Google Scholar 

  • Schnörr, C. 1994. Bewegungssegmentation von Bildfolgen durch die Minimierung konvexer nicht-quadratischer Funktionale. In Mustererkennung 1994, W. Kropatsch and H. Bischof (Ed.) Springer, Berlin, pp. 178–185.

    Google Scholar 

  • Schnörr, C., 1994. Segmentation of visual motion by minimizing convex non-quadratic functionals. In Proc. Twelfth International Conference on Pattern Recognition, Jerusalem, Israel, Oct. IEEE Computer Society Press, vol. A, pp. 661–663.

    Google Scholar 

  • Shulman, D. and Hervé, J. Regularization of discontinuous flow fields. In Proc. Workshop on Visual Motion, Irvine, CA, IEEE Computer Society Press, pp. 81–86.

    Google Scholar 

  • Szeliski, R. and Coughlan, J. 1994. Hierarchical spline-based image registration. In Proc. 1994 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Seattle, WA, June IEEE Computer Society Press, pp. 194–201.

    Google Scholar 

  • Tistarelli, M. 1994. Multiple constraints for optical flow. In Computer Vision – ECCV ’94, J.-O. Eklundh (Ed.) vol. 800 of Lecture Notes in Computer Science, Springer, Berlin, pp. 61–70.

    Google Scholar 

  • Uras, S., et al. 1988. A computational approach to motion perception. Biological Cybernetics, 60:79–87.

    Article  Google Scholar 

  • Weickert, J. and Schnörr, C. 2001. A theoretical framework for convex regularizers in PDE-based computation of imagemotion. International Journal of Computer Vision, 45(3):245–264.

    Article  MATH  Google Scholar 

  • Weickert, J. and Schnörr, C. 2001. Variational optic flow computation with a spatio-temporal smoothness constraint. Journal of Mathematical Imaging and Vision, 14(3):245–255.

    Article  MATH  Google Scholar 

  • Young, D. M. Iterative Solution of Large Linear Systems. Academic Press, New York, 1971.

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute for Mathematics, University of Lübeck, Wallstrasse 40, 23560, Lübeck, Germany

    Nils Papenberg

  2. Mathematical Image Analysis Group, Faculty of Mathematics and Computer Science, Saarland University, Building 27, 66041, Saarbrücken, Germany

    Andrés Bruhn, Thomas Brox, Stephan Didas & Joachim Weickert

Authors
  1. Nils Papenberg
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrés Bruhn
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thomas Brox
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stephan Didas
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Joachim Weickert
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nils Papenberg.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Papenberg, N., Bruhn, A., Brox, T. et al. Highly Accurate Optic Flow Computation with Theoretically Justified Warping. Int J Comput Vision 67, 141–158 (2006). https://doi.org/10.1007/s11263-005-3960-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-005-3960-y

Keywords

Search

Navigation