Spatiotemporal Closure

  • Alex Levinshtein
  • Cristian Sminchisescu
  • Sven Dickinson
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6492)


Spatiotemporal segmentation is an essential task for video analysis. The strong interconnection between finding an object’s spatial support and finding its motion characteristics makes the problem particularly challenging. Motivated by closure detection techniques in 2D images, this paper introduces the concept of spatiotemporal closure. Treating the spatiotemporal volume as a single entity, we extract contiguous “tubes” whose overall surface is supported by strong appearance and motion discontinuities. Formulating our closure cost over a graph of spatiotemporal superpixels, we show how it can be globally minimized using the parametric maxflow framework in an efficient manner. The resulting approach automatically recovers coherent spatiotemporal components, corresponding to objects, object parts, and object unions, providing a good set of multiscale spatiotemporal hypotheses for high-level video analysis.


Video Sequence Active Contour Graph Construction Motion Segmentation Shot Boundary 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Wang, D.: Unsupervised video segmentation based on watersheds and temporal tracking. CirSysVideo 8, 539–546 (1998)Google Scholar
  2. 2.
    Moscheni, F., Bhattacharjee, S., Kunt, M.: Spatiotemporal segmentation based on region merging. PAMI 20, 897–915 (1998)CrossRefGoogle Scholar
  3. 3.
    Gelgon, M., Bouthemy, P.: A region-level motion-based graph representation and labeling for tracking a spatial image partition. Pattern Recognition 33, 725–740 (2000)CrossRefGoogle Scholar
  4. 4.
    Deng, Y., Manjunath, B.: Unsupervised segmentation of color-texture regions in images and video. PAMI 23, 800–810 (2001)CrossRefGoogle Scholar
  5. 5.
    DeMenthon, D.: Spatio-temporal segmentation of video by hierarchical mean shift analysis. In: SMVP (2002)Google Scholar
  6. 6.
    Shi, J., Malik, J.: Motion segmentation and tracking using normalized cuts. In: ICCV, pp. 1154–1160 (1998)Google Scholar
  7. 7.
    Fowlkes, C., Belongie, S., Malik, J.: Efficient spatiotemporal grouping using the nyström method. In: CVPR, pp. 231–238 (2001)Google Scholar
  8. 8.
    Huang, Y., Liu, Q., Metaxas, D.: Video object segmentation by hypergraph cut. In: CVPR, pp. 1738–1745 (2009)Google Scholar
  9. 9.
    Greenspan, H., Goldberger, J., Mayer, A.: Probabilistic space-time video modeling via piecewise gmm. PAMI 26, 384–396 (2004)CrossRefGoogle Scholar
  10. 10.
    Levinshtein, A., Sminchisescu, C., Dickinson, S.: Optimal Contour Closure by Superpixel Grouping. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6312, pp. 480–493. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  11. 11.
    Levinshtein, A., Stere, A., Kutulakos, K.N., Fleet, D.J., Dickinson, S.J., Siddiqi, K.: Turbopixels: Fast superpixels using geometric flows. PAMI 31, 2290–2297 (2009)CrossRefGoogle Scholar
  12. 12.
    Carreira, J., Sminchisescu, C.: Constrained parametric min-cuts for automatic object segmentation. In: CVPR (2010)Google Scholar
  13. 13.
    Kolmogorov, V., Boykov, Y., Rother, C.: Applications of parametric maxflow in computer vision. In: ICCV (2007)Google Scholar
  14. 14.
    Li, F., Carreira, J., Sminchisescu, C.: Object Recognition as Ranking Holistic Figure-Ground Hypotheses. In: CVPR (2010)Google Scholar
  15. 15.
    Pritch, Y., Rav-Acha, A., Peleg, S.: Nonchronological video synopsis and indexing. PAMI 30, 1971–1984 (2008)CrossRefGoogle Scholar
  16. 16.
    Welch, G., Bishop, G.: An introduction to the kalman filter. Technical report (1995)Google Scholar
  17. 17.
    Black, M., Jepson, A.: Eigentracking: Robust matching and tracking of articulated objects using a view-based representation. IJCV 26, 63–84 (1998)CrossRefGoogle Scholar
  18. 18.
    Isard, M., Blake, A.: Condensation - conditional density propagation for visual tracking. IJCV 29, 5–28 (1998)CrossRefGoogle Scholar
  19. 19.
    Comaniciu, D., Ramesh, V., Meer, P.: Kernel-based object tracking. PAMI 25, 564–577 (2003)CrossRefGoogle Scholar
  20. 20.
    Megret, R., DeMenthon, D.: A survey of spatio-temporal grouping techniques. Technical report, University of Maryland, College Park (2002)Google Scholar
  21. 21.
    Wang, J., Adelson, E.: Representing moving images with layers. TIP 3, 625–638 (1994)Google Scholar
  22. 22.
    Weiss, Y., Adelson, E.H.: A unified mixture framework for motion segmentation: Incorporating spatial coherence and estimating the number of models. In: CVPR, p. 321 (1996)Google Scholar
  23. 23.
    Weiss, Y.: Smoothness in layers: Motion segmentation using nonparametric mixture estimation. In: CVPR, p. 520 (1997)Google Scholar
  24. 24.
    Jojic, N., Frey, B.J.: Learning flexible sprites in video layers. In: CVPR, vol. 1, p. 199 (2001)Google Scholar
  25. 25.
    Jepson, A.D., Fleet, D.J., Black, M.J.: A layered motion representation with occlusion and compact spatial support. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2350, pp. 692–706. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  26. 26.
    Bascle, B., Deriche, R.: Region tracking through image sequences. In: ICCV, p. 302 (1995)Google Scholar
  27. 27.
    Paragios, N., Deriche, R.: Geodesic active contours and level sets for the detection and tracking of moving objects. PAMI 22, 266–280 (2000)CrossRefGoogle Scholar
  28. 28.
    Chung, D., MacLean, W., Dickinson, S.: Integrating region and boundary information for spatially coherent object tracking. IVC 24, 680–692 (2006)CrossRefGoogle Scholar
  29. 29.
    Cremers, D., Soatto, S.: Motion competition: A variational approach to piecewise parametric motion segmentation. IJCV 62, 249–265 (2005)CrossRefGoogle Scholar
  30. 30.
    Patras, I., Lagendijk, R.L., Hendriks, E.A.: Video segmentation by map labeling of watershed segments. PAMI 23, 326–332 (2001)CrossRefGoogle Scholar
  31. 31.
    Martin, D.R., Fowlkes, C.C., Malik, J.: Learning to detect natural image boundaries using local brightness, color, and texture cues. PAMI 26, 530–549 (2004)CrossRefGoogle Scholar
  32. 32.
    Stein, A., Hoiem, D., Hebert, M.: Learning to find object boundaries using motion cues. In: ICCV (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Alex Levinshtein
    • 1
  • Cristian Sminchisescu
    • 2
  • Sven Dickinson
    • 1
  1. 1.University of TorontoCanada
  2. 2.University of BonnGermany

Personalised recommendations