International Journal of Computer Vision

, Volume 80, Issue 2, pp 242–259

Motion and Appearance Nonparametric Joint Entropy for Video Segmentation

  • Sylvain Boltz
  • Ariane Herbulot
  • Eric Debreuve
  • Michel Barlaud
  • Gilles Aubert
Article

Abstract

This paper deals with video segmentation based on motion and spatial information. Classically, the motion term is based on a motion compensation error (MCE) between two consecutive frames. Defining a motion-based energy as the integral of a function of the MCE over the object domain implicitly results in making an assumption on the MCE distribution: Gaussian for the square function and, more generally, parametric distributions for functions used in robust estimation. However, these assumptions are not necessarily appropriate. Instead, we propose to define the energy as a function of (an estimation of) the MCE distribution. This function was chosen to be a continuous version of the Ahmad-Lin entropy approximation, the purpose being to be more robust to outliers inherently present in the MCE. Since a motion-only constraint can fail with homogeneous objects, the motion-based energy is enriched with spatial information using a joint entropy formulation. The resulting energy is minimized iteratively using active contours. This approach provides a general framework which consists in defining a statistical energy as a function of a multivariate distribution, independently of the features associated with the object of interest. The link between the energy and the features observed or computed on the video sequence is then made through a nonparametric, kernel-based distribution estimation. It allows for example to keep the same energy definition while using different features or different assumptions on the features.

Keywords

Spatio-temporal segmentation Nonparametric distribution Joint entropy Active contour 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ahmad, I. A., & Lin, P. E. (1976). A nonparametric estimation of the entropy for absolutely continuous distributions. IEEE Transactions on Information Theory, 22(3), 372–375. MATHCrossRefMathSciNetGoogle Scholar
  2. Alvarez, L., Weickert, J., & Sánchez, J. (1999). A scale-space approach to nonlocal optical flow calculations. In International conference on scale-space theories in computer vision, Corfu, Greece. Google Scholar
  3. Awate, S. P., & Whitaker, R. T. (2006). Unsupervised, information-theoretic, adaptive image filtering for image restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(3), 364–376. CrossRefGoogle Scholar
  4. Aubert, G., Barlaud, M., Faugeras, O., & Jehan-Besson, S. (2003). Image segmentation using active contours: Calculus of variations or shape gradients? SIAM Journal on Applied Mathematics, 1(2), 2128–2145. CrossRefMathSciNetGoogle Scholar
  5. Black, M. J., & Anandan, P. (1996). The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. Computer Vision and Image Understanding, 63(1), 75–104. CrossRefGoogle Scholar
  6. Boltz, S., Herbulot, A., Debreuve, E., & Barlaud, M. (2006). Entropy-based space-time segmentation in video sequences. In ECCV workshop on statistical methods in multi-image and video processing, Graz, Austria. Google Scholar
  7. Boltz, S., Debreuve, E., & Barlaud, M. (2007). High-dimensional statistical distance for region-of-interest tracking: Application to combining a soft geometric constraint with radiometry. In International conference on computer vision and pattern recognition, Minneapolis (MN), USA. Google Scholar
  8. Brox, T., Rousson, M., Deriche, R., & Weickert, J. (2003). Unsupervised segmentation incorporating colour, texture, and motion. In International conference on computer analysis of images and patterns, Groningen, The Netherlands. Google Scholar
  9. Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In European conference on computer vision, Prague, Czech Republic. Google Scholar
  10. Bruhn, A., Weickert, J., & Schnörr, C. (2002). Combining the advantages of local and global optic flow methods. In DAGM symposium on pattern recognition, Zurich, Switzerland. Google Scholar
  11. Caselles, V., Kimmel, R., & Sapiro, G. (1997). Geodesic active contours. International Journal of Computer Vision, 22(1), 61–79. MATHCrossRefGoogle Scholar
  12. Charbonnier, P., Blanc-Féraud, L., Aubert, G., & Barlaud, M. (1997). Deterministic edge-preserving regularization in computed imaging. IEEE Transactions on Image Processing, 6(2), 298–311. CrossRefGoogle Scholar
  13. Comaniciu, D., Ramesh, V., & Meer, P. (2000). Real-time tracking of non-rigid objects using mean shift. In International conference on computer vision and pattern recognition, Hilton Head Island (SC), USA. Google Scholar
  14. Cremers, D., & Soatto, S. (2003). Variational space-time motion segmentation. In International conference on computer vision, Nice, France. Google Scholar
  15. Cremers, D., & Soatto, S. (2005). Motion competition: A variational framework for piecewise parametric motion segmentation. International Journal of Computer Vision, 62(3), 249–265. CrossRefGoogle Scholar
  16. Cremers, D., Osher, S. J., & Soatto, S. (2006). Kernel density estimation and intrinsic alignment for shape priors in level set segmentation. International Journal of Computer Vision, 69(3), 335–351. CrossRefGoogle Scholar
  17. Cremers, D., Rousson, M., & Deriche, R. (2007). A review of statistical approaches to level set segmentation: Integrating color, texture, motion and shape. International Journal of Computer Vision, 72(2), 195–215. CrossRefGoogle Scholar
  18. Delfour, M. C., & Zolésio, J.-P. (2001). Shapes and geometries: Analysis, differential calculus and optimization. Advances in design and control. Philadelphia: Society for Industrial and Applied Mathematics. MATHGoogle Scholar
  19. Dórea, C. C., Pardàs, M., & Marqués, F. (2006). Generation of long-term color and motion coherent partitions. In International conference on image processing, Atlanta (GA), USA. Google Scholar
  20. Elgammal, A., Duraiswami, R., & Davis, L. S. (2003). Probabilistic tracking in joint feature-spatial spaces. In International conference on computer vision and pattern recognition, Madison (WI), USA. Google Scholar
  21. Freedman, D., & Zhang, T. (2004). Active contours for tracking distributions. IEEE Transactions on Image Processing, 13(4), 518–526. CrossRefGoogle Scholar
  22. Fukunaga, K. (1990). Introduction to statistical pattern recognition (2nd ed.). San Diego: Academic Press. MATHGoogle Scholar
  23. Goria, M. N., Leonenko, N. N., Mergel, V. V., & Novi Inverardi, P. L. (2005). A new class of random vector entropy estimators and its applications in testing statistical hypotheses. Journal of Nonparametric Statistics, 17(3), 277–297. MATHCrossRefMathSciNetGoogle Scholar
  24. Herbulot, A., Boltz, S., Debreuve, E., & Barlaud, M. (2006). Robust motion-based segmentation in video sequences using entropy estimator. In International conference on image processing, Atlanta (GA), USA. Google Scholar
  25. Hintermuller, M., & Ring, W. (2004). A second order shape optimization approach for image segmentation. SIAM Journal on Applied Mathematics, 64(2), 442–467. CrossRefMathSciNetGoogle Scholar
  26. Jehan-Besson, S., Barlaud, M., & Aubert, G. (2003). DREAM2S: deformable regions driven by an Eulerian accurate minimization method for image and video segmentation. International Journal of Computer Vision, 53(1), 45–70. CrossRefGoogle Scholar
  27. Kim, J., Fisher, J. W. F., Yezzi, A., Çetin, M., & Willsky, A. S. (2005). A nonparametric statistical method for image segmentation using information theory and curve evolution. IEEE Transactions on Image Processing, 14(10), 1486–1502. CrossRefGoogle Scholar
  28. Leventon, M., Grimson, E., & Faugeras, O. (2000). Statistical shape influence in geodesic active contour. In International conference on computer vision and pattern recognition, Hilton Head Island (SC), USA. Google Scholar
  29. Leventon, M. E., Faugeras, O., Grimson, W. E. L., & Wells III, W. M. (2000). Level set based segmentation with intensity and curvature priors. In Workshop on mathematical methods in biomedical image analysis, Hilton Head Island (SC), USA. Google Scholar
  30. Lucas, B. D., & Kanade, T. (1981). An iterative image registration technique with an application to stereo vision. In International joint conference on artificial intelligence, Vancouver, Canada. Google Scholar
  31. Mittal, A., & Paragios, N. (2004). Motion-based background subtraction using adaptive kernel density estimation. In International conference on computer vision and pattern recognition, Washington, DC, USA. Google Scholar
  32. Neemwuchwala, H., & Hero, A. O. (2005). Entropic graphs for registration. In R. S. Blum & Z. Liu (Eds.), Multi-sensor image fusion and its applications (Chap. 6, pp. 185–235). New York: Dekker. Google Scholar
  33. Odobez, J.-M., & Bouthemy, P. (1995). Robust multiresolution estimation of parametric motion models. Journal of Visual Communication and Image Representation, 6(4), 348–365. CrossRefGoogle Scholar
  34. Paragios, N., & Deriche, R. (2002). Geodesic active regions: A new framework to deal with frame partition problems in computer vision. Journal of Visual Communication and Image Representation, 13(1–2), 249–268. CrossRefGoogle Scholar
  35. Parzen, E. (1962). On the estimation of a probability density function and mode. Annals of Mathematical Statistics, 33(3), 1065–1076. MATHCrossRefMathSciNetGoogle Scholar
  36. Rubner, Y., Tomasi, C., & Guibas, L. J. (2000). The earth mover’s distance as a metric for image retrieval. International Journal of Computer Vision, 40(2), 99–121. MATHCrossRefGoogle Scholar
  37. Samson, C., Blanc-Féraud, L., Aubert, G., & Zerubia, J. (2000). A level set model for image classification. International Journal of Computer Vision, 40(3), 187–197. MATHCrossRefGoogle Scholar
  38. Schoenemann, T., & Cremers, D. (2006). Near real-time motion segmentation using graph cuts. In DAGM symposium on pattern recognition, Berlin, Germany. Google Scholar
  39. Scott, D. W. (1992). Multivariate density estimation: theory, practice, and visualization. New York: Wiley. MATHGoogle Scholar
  40. Silverman, B. W. (1986). Density estimation for statistics and data analysis. London: Chapman and Hall. MATHGoogle Scholar
  41. Weickert, J., & Schnörr, C. (2001). Variational optic flow computation with a spatio-temporal smoothness constraint. Journal of Mathematical Imaging and Vision, 14(3), 245–255. MATHCrossRefGoogle Scholar
  42. Wells, W., Viola, P., Atsumi, H., Nakajima, S., & Kikinis, R. (1996). Multi-modal volume registration by maximization of mutual information. Medical Image Analysis, 1(1), 35–51. CrossRefGoogle Scholar
  43. Wu, S. F., & Kittler, J. (1993). A gradient-based method for general motion estimation and segmentation. Journal of Visual Communication and Image Representation, 4(1), 25–38. CrossRefGoogle Scholar
  44. Yezzi, A., Tsai, A., & Willsky, A. (2002). A fully global approach to image segmentation via coupled curve evolution equations. Journal of Visual Communication and Image Representation, 13(1), 195–216. CrossRefGoogle Scholar
  45. Zhu, S., & Ma, K.-K. (2000). A new diamond search algorithm for fast block-matching motion estimation. IEEE Transactions on Image Processing, 9(2), 287–290. CrossRefMathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  • Sylvain Boltz
    • 1
  • Ariane Herbulot
    • 1
  • Eric Debreuve
    • 1
  • Michel Barlaud
    • 1
  • Gilles Aubert
    • 2
  1. 1.Laboratoire I3SUniversité de Nice-Sophia Antipolis/CNRSNiceFrance
  2. 2.Laboratoire J.-A. DieudonnéUniversité de Nice-Sophia Antipolis/CNRSNiceFrance

Personalised recommendations