Skip to main content
SpringerLink
Log in

Motion Competition: A Variational Approach to Piecewise Parametric Motion Segmentation

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

We present a novel variational approach for segmenting the image plane into a set of regions of parametric motion on the basis of two consecutive frames from an image sequence. Our model is based on a conditional probability for the spatio-temporal image gradient, given a particular velocity model, and on a geometric prior on the estimated motion field favoring motion boundaries of minimal length.

Exploiting the Bayesian framework, we derive a cost functional which depends on parametric motion models for each of a set of regions and on the boundary separating these regions. The resulting functional can be interpreted as an extension of the Mumford-Shah functional from intensity segmentation to motion segmentation. In contrast to most alternative approaches, the problems of segmentation and motion estimation are jointly solved by continuous minimization of a single functional. Minimizing this functional with respect to its dynamic variables results in an eigenvalue problem for the motion parameters and in a gradient descent evolution for the motion discontinuity set.

We propose two different representations of this motion boundary: an explicit spline-based implementation which can be applied to the motion-based tracking of a single moving object, and an implicit multiphase level set implementation which allows for the segmentation of an arbitrary number of multiply connected moving objects.

Numerical results both for simulated ground truth experiments and for real-world sequences demonstrate the capacity of our approach to segment objects based exclusively on their relative motion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Ayer, S. and Sawhney, H.S. 1995. Layered representation of motion video using robust maximum likelihood estimation of mixture models and MDL encoding. In Proc. of the Int. Conf. on Comp. Vis., Boston, USA, pp. 777–784.

  • BigBigünn, J., Granlund, G.H., and Wiklund, J. 1991. Multidimensional orientation estimation with applications to texture analysis and optical flow. IEEE PAMI, 13(8):775–790.

    Google Scholar 

  • Black, M.J. 1994. Recursive non-linear estimation of discontinuous flow fields. In Proc. of the Europ. Conf. on Comp. Vis., J.O. Eklundh, (Ed.), vol. 800 of LNCS, Springer-Verlag, pp. 138–145.

  • Black, M.J. and Anandan, P. 1996. The robust estimation of multiple motions: Parametric and piecewise–smooth flow fields. Comp. Vis. Graph. Image Proc.: IU, 63(1):75–104.

    Google Scholar 

  • Bouthemy, P. and Francois, E. 1993. Motion segmentation and qualitative dynamic scene analysis from an image sequence. Int. J. of Computer Vision, 10(2):157–182.

    Article  Google Scholar 

  • Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In European Conf. on Computer Vision, T. Pajdla and V. Hlavac (Eds.), vol. 3024 of LNCS, Springer, Prague, pp. 25–36.

  • Caselles, V. and Coll, B. 1996. Snakes in movement. SIAM J. Numer. Anal., 33:2445–2456.

    Article  Google Scholar 

  • Caselles, V., Kimmel, R., and Sapiro, G. 1995. Geodesic active contours. In Proc. IEEE Intl. Conf. on Comp. Vis., Boston, USA, pp. 694–699.

  • Chan, T. and Vese, L. 2001. Active contours without edges. IEEE Trans. Image Processing, 10(2):266–277.

    Article  Google Scholar 

  • Cremers, D. 2003a. A multiphase level set framework for variational motion segmentation. In Int. Conf. on Scale Space Theories in Computer Vision, L. Griffith (Ed.), vol. 2695 of LNCS, Isle of Skye, Springer, pp. 599–614.

  • Cremers, D. 2003b. A variational framework for image segmentation combining motion estimation and shape regularization. In IEEE Conf. on Comp. Vis. and Patt. Recog., C. Dyer and P. Perona (Eds.), vol. 1, pp. 53–58.

  • Cremers, D. and SchnSchnörrrr, C. 2003. Statistical shape knowledge in variational motion segmentation. Image and Vision Computing, 21(1):77–86.

    Article  Google Scholar 

  • Cremers, D. and Soatto, S. 2003. Variational space-time motion segmentation. In IEEE Int. Conf. on Computer Vision, B. Triggs and A. Zisserman (Eds.), Nice, vol. 2, pp. 886–892.

  • Cremers, D., Tischhäuser, F., Weickert, J., and Schnörr, C. 2002. Diffusion Snakes: Introducing statistical shape knowledge into the Mumford–Shah functional. Int. J. of Computer Vision, 50(3):295–313.

    Article  Google Scholar 

  • Cremers, D. and Yuille, A.L. 2003. A generative model based approach to motion segmentation. In Pattern Recognition, B. Michaelis and G. Krell (Eds.), vol. 2781 of LNCS, Springer, Magdeburg, pp. 313–320.

    Google Scholar 

  • Delingette, H. and Montagnat, J. 2000. New algorithms for controlling active contours shape and topology. In Proc. of the Europ. Conf. on Comp. Vis., D. Vernon (Ed.), vol. 1843 of LNCS, Springer, pp. 381–395.

  • Farnebäck, G. 1999. Spatial Domain Methods for Orientation and Velocity Estimation. PhD thesis, Dept. of Electrical Engineering, Linköpings universitet.

  • Farnebäck, G. 2001. Very high accuracy velocity estimation using orientation tensors, parametric motion, and segmentation of the motion field. In ICCV, vol. 1, pp. 171–177.

    Google Scholar 

  • Gomes, J. and Faugeras, O.D. 2000. Level sets and distance functions. In Proc. of the Europ. Conf. on Comp. Vis., D. Vernon (Ed.), vol. 1842 of LNCS, Dublin, Ireland, Springer, pp. 588–602.

  • Heitz, F. and Bouthemy, P. 1993. Multimodal estimation of discontinuous optical flow using markov random fields. IEEE PAMI, 15(12):1217–1232.

    Google Scholar 

  • Horn, B.K.P. and Schunck, B.G. 1981. Determining optical flow. A.I., 17:185–203.

    Google Scholar 

  • Jehan-Besson, S., Barlaud, M., and Aubert, G. 2003. DREAM2S: Deformable regions driven by an eulerian accurate minimization method for image and video segmentation. Int. J. of Computer Vision, 53(1):45–70.

    Article  Google Scholar 

  • Jepson, A. and Black, M.J. 1993. Mixture models for optic flow computation. In Proc. IEEE Conf. on Comp. Vision Patt. Recog., New York, pp. 760–761.

  • Kass, M., Witkin, A., and Terzopoulos, D. 1988. Snakes: Active contour models. Int. J. of Computer Vision, 1(4):321–331.

    Article  Google Scholar 

  • Kichenassamy, S., Kumar, A., Olver, P.J., Tannenbaum, A., and Yezzi, A.J. 1995. Gradient flows and geometric active contour models. In Proc. IEEE Intl. Conf. on Comp. Vis., Boston, USA, pp. 810–815.

  • Konrad, J. and Dubois, E. 1992. Bayesian estimation of motion vector fields. IEEE PAMI, 14(9):910–927.

    Google Scholar 

  • Kornprobst, P., Deriche, R., and Aubert, G. 1999. Image sequence analysis via partial differential equations. J. Math. Im. Vis., 11(1):5–26.

    Article  Google Scholar 

  • Lie, J., Lysaker, M., and Tai, X.-C. 2003. A variant of the level set method and applications to image segmentation. Technical Report 03–50, Computational Applied Mathematics, UCLA, Los Angeles.

  • Lucas, B.D. and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In Proc.7th International Joint Conference on Artificial Intelligence, Vancouver, pp. 674–679.

  • Malladi, R., Sethian, J.A., and Vemuri, B.C. 1995. Shape modeling with front propagation: A level set approach. IEEE PAMI, 17(2):158–175.

    Google Scholar 

  • McInerney, T. and Terzopoulos, D. 1995. Topologically adaptable snakes. In Proc. 5th Int. Conf. on Computer Vision, IEEE Comp. Soc. Press: Los Alamitos, California, pp. 840–845.

  • Memin, E. and Perez, P. 1998. Dense estimation and object-based segmentation of the optical flow with robust techniques. IEEE Trans. on Im. Proc., 7(5):703–719.

    Article  Google Scholar 

  • Memin, E. and Perez, P. 2002. Hierarchical estimation and segmentation of dense motion fields. Int. J. of Computer Vision, 46(2):129–155.

    Article  Google Scholar 

  • Morel, J.-M. and Solimini, S. 1995. Variational Methods in Image Segmentation. Birkhäuser: Boston.

    Google Scholar 

  • Mumford, D. and Shah, J. 1989. Optimal approximations by piecewise smooth functions and associated variational problems. Comm. Pure Appl. Math., 42:577–685.

    Google Scholar 

  • Nagel, H.H. and Enkelmann, W. 1986. An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE PAMI, 8(5):565–593.

    Google Scholar 

  • Nestares, O., Fleet, D.J., and Heeger, D.J. 2000. Likelihood functions and confidence bounds for total-least-squares problems. In Proc. Conf. Computer Vis. and Pattern Recog., vol. 1, Hilton Head Island, SC, pp. 760–767.

  • Odobez, J.-M. and Bouthemy, P. 1995. Robust multiresolution estimation of parametric motion models. J. of Visual Commun. and Image Repr., 6(4):348–365.

    Article  Google Scholar 

  • Odobez, J.-M. and Bouthemy, P. 1998. Direct incremental model-based image motion segmentation for video analysis. Signal Proc., 66:143–155.

    Article  Google Scholar 

  • Osher, S.J. and Sethian, J.A. 1988. Fronts propagation with curvature dependent speed: Algorithms based on Hamilton–Jacobi formulations. J. of Comp. Phys., 79:12–49.

    Article  Google Scholar 

  • Paragios, N. and Deriche, R. 2000. Geodesic active contours and level sets for the detection and tracking of moving objects. IEEE PAMI, 22(3):266–280.

    Google Scholar 

  • Samson, C., Blanc-FBlanc-Féraudraud, L., Aubert, G., and Zerubia, J. 2000. A level set model for image classification. Int. J. of Computer Vision, 40(3):187–197.

    Article  Google Scholar 

  • Schnörr, C. 1992. Computation of discontinuous optical flow by domain decomposition and shape optimization. Int. J. of Computer Vision, 8(2):153–165.

    Article  Google Scholar 

  • Shi, J. and Malik, J. 1998. Motion segmentation and tracking using normalized cuts. In Intl. Conf. on Comp. Vision, Bombay, India.

  • Sussman, M. and Fatemi, E. 1999. An efficient, interface-preserving level set redistancing algorithm and its application to interfacial incompressible fluid flow. SIAM J. Sci. Comput., 20(4):1165–1191.

    Article  Google Scholar 

  • Sussman, M., Smereka P., and Osher, S.J. 1994. A level set approach for computing solutions to incompressible twophase flow. J. of Comp. Phys., 94:146–159.

    Article  Google Scholar 

  • Unal, G., Krim, H., and Yezzi, A. (to appear). Fast incorporation of optic flow into active polygons. IEEE Trans. on Image Processing.

  • Wang, J.Y.A. and Adelson, E.H. 1994. Representating moving images with layers. IEEE Trans. on Image Processing, 3(5):625–638.

    Article  Google Scholar 

  • Weickert, J. and Schnörr, C. 2001. A theoretical framework for convex regularizers in PDE–based computation of image motion. Int. J. of Computer Vision, 45(3):245–264.

    Article  Google Scholar 

  • Weiss, Y. 1997. Smoothness in layers: Motion segmentation using nonparametric mixture estimation. In Proc. IEEE Conf. on Comp. Vision Patt. Recog., Puerto Rico, pp. 520–527.

  • Weiss, Y. and Fleet, D.J. 2001. Velocity likelihoods in biological and machine vision. In Probabilistic Models of the Brain: Perception and Neural Function, M.S. Lewicki, R.P.N. Rao, and B.A. Olshausen (Eds.), MIT Press, pp. 81–100.

  • Yezzi, A. and Soatto, S. 2003. Stereoscopic segmentation. Int. J. of Computer Vision, 53(1):31–43.

    Article  Google Scholar 

  • Zhao, H.-K., Chan, T., Merriman, B., and Osher, S. 1996. A variational level set approach to multiphase motion. J. of Comp. Phys., 127:179–195.

    Article  Google Scholar 

  • Zheng, H. and Blostein, S.D. 1995. Motion-based object segmentation and estimation. IEEE Trans. on Image Processing, 4(9):1225–1235.

    Article  Google Scholar 

  • Zhu, S.C. and Yuille, A. 1996. Region competition: Unifying snakes, region growing, and Bayes/MDL for multiband image segmentation. IEEE PAMI, 18(9):884–900.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, University of California, Los Angeles, CA, 90095, USA

    Daniel Cremers & Stefano Soatto

  2. Siemens Corporate Research, Princeton, NJ

    Daniel Cremers

Authors
  1. Daniel Cremers
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Stefano Soatto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Daniel Cremers.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Cremers, D., Soatto, S. Motion Competition: A Variational Approach to Piecewise Parametric Motion Segmentation. Int J Comput Vision 62, 249–265 (2005). https://doi.org/10.1007/s11263-005-4882-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-005-4882-4

Keywords

Search

Navigation