Abstract
Many differential methods for the recovery of the optic flow field from an image sequence can be expressed in terms of a variational problem where the optic flow minimizes some energy. Typically, these energy functionals consist of two terms: a data term, which requires e.g. that a brightness constancy assumption holds, and a regularizer that encourages global or piecewise smoothness of the flow field. In this paper we present a systematic classification of rotation invariant convex regularizers by exploring their connection to diffusion filters for multichannel images. This taxonomy provides a unifying framework for data-driven and flow-driven, isotropic and anisotropic, as well as spatial and spatio-temporal regularizers. While some of these techniques are classic methods from the literature, others are derived here for the first time. We prove that all these methods are well-posed: they posses a unique solution that depends in a continuous way on the initial data. An interesting structural relation between isotropic and anisotropic flow-driven regularizers is identified, and a design criterion is proposed for constructing anisotropic flow-driven regularizers in a simple and direct way from isotropic ones. Its use is illustrated by several examples.
Similar content being viewed by others
References
Alvarez, L., Esclarín, J., Lefébure, M., and Sánchez, J. 1999. A PDE model for computing the optical flow. In Proc. XVI Congreso de Ecuaciones Diferenciales y Aplicaciones, C.E.D.Y.A. XVI, Las Palmas de Gran Canaria, Sept. 21–24, 1999, pp. 1349–1356.
Alvarez, L., Weickert, J., and Sánchez, J. 2000. Reliable estimation of dense optical flow fields with large displacements. Int. J. Comput. Vision, 39:41–56.
Anandan, P. 1989. A computational framework and an algorithm for the measurement of visual motion. Int. J. Comput. Vision, 2:283–310.
Aubert, G., Deriche, R., and Kornprobst, P. 1999. Computing optical flow via variational techniques. SIAM J. Appl. Math, 60:156–182.
Barron, J.L., Fleet, D.J., and Beauchemin, S.S. 1994. Performance of optical flow techniques. Int. J. Comput. Vision, 12:43–77.
Bertero, M., Poggio, T.A., and Torre, V. 1988. Ill-posed problems in early vision. Proc. IEEE, 76:869–889.
Black, M.J. and Anandan, P. 1991. Robust dynamic motion estimation over time. In Proc. IEEE Comp. Soc. Conf. on Computer Vision and Pattern Recognition, CVPR ‘91, Maui, June 3–6, 1991, IEEE Computer Society Press: Los Alamitos, pp. 292–302.
Black, M.J., and Anandan, P. 1996. The robust estimation of multiple motions: Parametric and piecewise smooth flow fields. Computer Vision and Image Understanding, 63:75–104.
Blake, A. and Zisserman, A. 1987.Visual Reconstruction. MIT Press: Cambridge (Mass.).
Blanc-Féraud, L., Barlaud, M., and Gaidon, T. 1993. Motion estimation involving discontinuities in a multiresolution scheme. Optical Engineering, 32(7):1475–1482.
Charbonnier, P., Blanc-Féraud, L., Aubert, G., and Barlaud, M. 1994. Two deterministic half-quadratic regularization algorithms for computed imaging. In Proc. IEEE Int. Conf. Image Processing, ICIP-94, Austin, Nov. 13–16, 1994, Vol. 2, IEEE Computer Society Press: Los Alamitos, pp. 168–172.
Cohen, I. 1993. Nonlinear variational method for optical flow computation. In Proc. Eighth Scandinavian Conf. on Image Analysis, SCIA ‘93, Tromsø, May 25–28, 1993, Vol. 1, pp. 523–530.
Courant, R. and Hilbert, D. 1953. Methods of Mathematical Physics, Vol. 1. Interscience: New York.
Deriche, R., Kornprobst, P., and Aubert, G. 1995. Optical-flow estimation while preserving its discontinuities: A variational approach. In Proc. Second Asian Conf. Computer Vision, ACCV ‘95, Singapore, Dec. 5–8, 1995, Vol. 2, pp. 290–295.
Di Zenzo, S. 1986. A note on the gradient of a multi-image. Computer Vision, Graphics, and Image Processing, 33:116–125.
Elsgolc, L.E. 1961. Calculus of Variations. Pergamon: Oxford.
Enkelmann, W. 1988. Investigation of multigrid algorithms for the estimation of optical flow fields in image sequences. Computer Vision, Graphics and Image Processing, 43:150–177.
Finkbeiner, D.T. 1966. Introduction to Matrices and Linear Transformations. Freeman: San Francisco.
Fleet, D.J. and Jepson, A.D. 1990. Computation of component image velocity from local phase information. Int. J. Comput. Vision, 5:77–104.
Förstner, W. and Gülch, E. 1987. A fast operator for detection and precise location of distinct points, corners and centres of circular features. In Proc. ISPRS Intercommission Conf. on Fast Processing of Photogrammetric Data, Interlaken, June 2–4, 1987, pp. 281–305.
Galvin, B., McCane, B., Novins, K., Mason, D., and Mills, S. 1998. Recovering motion fields: An analysis of eight optical flow algorithms. In Proc. 1998 British Machine Vision Conference, BMVC '98, Southampton, Sept. 14–17, 1998.
Gerig, G., Kübler, O., Kikinis, R., and Jolesz, F.A. 1992. Nonlinear anisotropic filtering of MRI data. IEEE Trans. Medical Imaging, 11:221–232.
ter Haar Romeny, B., Florack, L., Koenderink, J., and Viergever, M. (Eds.). 1997. Scale-space Theory in Computer Vision. Springer: Berlin. Lecture Notes in Computer Science, Vol. 1252.
Heitz, F. and Bouthemy, P. 1993. Multimodal estimation of discontinuous optical flow using Markov random fields. IEEE Trans. Pattern Anal. Mach. Intell., 15:1217–1232.
Hinterberger, W. 1999. Generierung eines Films zwischen zwei Bildern mit Hilfe des optischen Flusses. M.Sc. thesis, Industrial Mathematics Institute, University of Linz, Austria.
Horn, B. and Schunck, B. 1981. Determining optical flow. Artificial Intelligence, 17:185–203.
Kimmel, R., Malladi, R., and Sochen, N. 2000. Images as embedded maps and minimal surfaces: Movies, color, texture, and volumetric medical images. Int. J. Comput. Vision, 39:111–129.
Koenderink, J.J. 1975. Invariant properties of the motion parallax field due to the movement of rigid bodies relative to an observer. Optica Acta, 22:773–791.
Kreyszig, E. 1959. Differential Geometry. University of Toronto Press: Toronto.
Kumar, A., Tannenbaum, A.R., and Balas, G.J. 1996. Optic flow: A curve evolution approach. IEEE Trans. Image Proc., 5:598–610.
van Laarhoven, P.J.M. and Aarts, E.H.L. 1988. Simulated Annealing: Theory and Applications. Reidel: Dordrecht.
Mémin, E. and Pérez, P. 1998. Dense estimation and object-based segmentation of the optical flow with robust techniques. IEEE Trans. Image Proc, 7:703–719.
Mitiche, A. and Bouthemy, P. 1996. Computation and analysis of image motion: A synopsis of current problems and methods. Int. J. Comput. Vision, 19:29–55.
Murray, D.W. and Buxton, B.F. 1987. Scene segmentation from visual motion using global optimization. IEEE Trans. Pattern Anal. Mach. Intell., 9:220–228.
Nagel, H.H. 1983. Constraints for the estimation of displacement vector fields from image sequences. In Proc. Eighth Int. Joint Conf. on Artificial Intelligence, IJCAI ‘83, Karlsruhe, Aug. 8–12, 1983, pp. 945–951.
Nagel, H.-H. 1987. On the estimation of optical flow: Relations between different approaches and some new results. Artificial Intelligence, 33:299–324.
Nagel, H.-H. 1990. Extending the ‘oriented smoothness constraint’ into the temporal domain and the estimation of derivatives of optical flow. In '90, O. Faugeras (Ed.). Lecture Notes in Computer Science, Vol. 427. Springer: Berlin, pp. 139–148.
Nagel, H.-H. and Enkelmann, W. 1986. An investigation of smoothness constraints for the estimation of displacement vector fields from images sequences. IEEE Trans. Pattern Anal. Mach. Intell., 8:565–593.
Nesi, P. 1993. Variational approach to optical flow estimation managing discontinuities. Image and Vision Computing, 11:419–439.
Nielsen, M., Johansen, P., Olsen, O.F., and Weickert, J. (Eds.). 1999. Scale-Space Theories in Computer Vision. Springer: Berlin. Lecture Notes in Computer Science, Vol. 1682.
Otte, M. and Nagel, H.-H. 1995. Estimation of optical flow based on higher-order spatiotemporal derivatives in interlaced and noninterlaced image sequences. Artificial Intelligence, 78:5–43.
Proesmans, M., Van Gool, L., Pauwels, E., and Oosterlinck, A. 1994.Determination of optical flow and its discontinuities using nonlinear diffusion. In '94, J.-O. Eklundh (Ed.). Springer: Berlin, pp. 295–304. Lecture Notes in Computer Science, Vol. 801.
Schnörr, C. 1991. Determining optical flow for irregular domains by minimizing quadratic functionals of a certain class. Int. J. Comput. Vision, 6:25–38.
Schnörr, C. 1993. On functionals with greyvalue-controlled smoothness terms for determining optical flow. IEEE Trans. Pattern Anal. Mach. Intell., 15:1074–1079.
Schnörr, C. 1994. Segmentation of visual motion by minimizing convex non-quadratic functionals. In Proc. 12th Int. Conf. Pattern Recognition, ICPR 12, Jerusalem, Oct. 9–13, 1994, Vol. A, IEEE Computer Society Press: Los Alamitos, pp. 661–663.
Schnörr, C. 1996. Convex variational segmentation of multi-channel images. In '96: Images,Wavelets and PDEs, M.-O. Berger, R. Deriche, I. Herlin, J. Jaffré, and J.-M. Morel (Eds.). Springer: London, pp. 201–207. Lecture Notes in Control and Information Sciences, Vol. 219.
Schnörr, C. and Sprengel, R. 1994. A nonlinear regularization approach to early vision. Biol. Cybernetics, 72:141–149.
Schnörr, C., Sprengel, R., and Neumann, B. 1996. A variational approach to the design of early vision algorithms. Computing, Suppl., 11:149–165.
Schnörr, C. and Weickert, J. 2000. Variational motion computation: Theoretical framework, problems and perspectives. In Mustererkennung 2000, G. Sommer, N. Krüger, and C. Perwass (Eds.). Springer: Berlin, pp. 476–487.
Snyder, M.A. 1991. On the mathematical foundations of smoothness constraints for the determination of optical flow and for surface reconstruction. IEEE Trans. Pattern Anal. Mach. Intell., 13:1105–1114.
Sporring, J., Nielsen, M., Florack, L., and Johansen, P. (Eds.). 1997.Gaussian Scale-Space Theory, Kluwer: Dordrecht.
Stevenson, R.L., Schmitz, B.E., and Delp, E.J. 1994. Discontinuity preserving regularization of inverse visual problems. IEEE Trans. Systems, Man and Cybernetics, 24:455–469.
Stiller, C. and Konrad, J. 1999. Estimating motion in image sequences. IEEE Signal Proc. Magazine, 16:70–91.
Weber, J. and Malik, J. 1995. Robust computation of optical flow in a multi-scale differential framework. Int. J. Comput. Vision, 14:67–81.
Weickert, J. 1994. Scale-space properties of nonlinear diffusion filtering with a diffusion tensor. Report No. 110, Laboratory of Technomathematics, University of Kaiserslautern, P.O. Box 3049, 67653 Kaiserslautern, Germany.
Weickert, J. 1998a. Anisotropic Diffusion in Image Processing. Teubner: Stuttgart.
Weickert, J. 1998b. On discontinuity-preserving optic flow. In Proc. Computer Vision and Mobile Robotics Workshop, CVMR '98, Santorini, Sept. 17–18, 1998, S. Orphanoudakis, P. Trahanias, J. Crowley, and N. Katevas (Eds.). pp. 115–122.
Weickert, J. 1999. Coherence-enhancing diffusion of colour images. Image and Vision Computing, 17:199–210.
Weickert, J. and Schnörr, C. 2001. Variational optic flow computation with a spatio-temporal smoothness constraint. J. Math. Imag. Vision, 14:245–255.
Whitaker, R. and Gerig, G. 1994.Vector-valued diffusion. Geometry-Driven Diffusion in ComputerVision, B.M. ter Haar Romeny (Ed.). Kluwer: Dordrecht, pp. 93–134.
Zeidler, E. 1990. Nonlinear Functional Analysis and its Applications, Vol. IIb. Springer: Berlin.
Ziemer, W.P. 1989. Weakly Differentiable Functions. Springer: New York.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Weickert, J., Schnörr, C. A Theoretical Framework for Convex Regularizers in PDE-Based Computation of Image Motion. International Journal of Computer Vision 45, 245–264 (2001). https://doi.org/10.1023/A:1013614317973
Issue Date:
DOI: https://doi.org/10.1023/A:1013614317973