Interactive Video Layer Decomposition and Matting

  • Yanli Li
  • Zhong Zhou
  • Wei Wu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6469)


The problem of accurate video layer decomposition is of vital importance in computer vision. Previous methods mainly focus on the foreground extraction. In this paper, we present a user-assisted framework to decompose videos and extract all layers, which is built on the depth information and over-segmented patches. The task is split into two stages: i) the clustering of over-segmented patches; ii) the propagation of layers along the video. Correspondingly, this paper has two contributions: i) a video decomposition method based on greedy over-segmented patches merging; ii) a layer propagation method via iteratively updating color Gaussian Mixture Models(GMM). We test this algorithm on real videos and verify that it outperforms state-of-the-art methods.


Gaussian Mixture Model Depth Information Stereo Match Video Object Motion Segmentation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Darrell, T., Pentland, A.: Cooperative robust estimation using layers of support. IEEE Trans. on Pattern Analysis and Machine Intelligence 17, 474–487 (1991)CrossRefGoogle Scholar
  2. 2.
    Wang, J., Adelson, E.: Representing moving images with layers. IEEE Trans. on Image Processing Special Issue: Image Sequence Compression 3, 625–638 (1994)CrossRefGoogle Scholar
  3. 3.
    Ke, Q., Kanade, T.: Robust subspace clustering by combined use of knnd metric and svd algorithm. In: CVPR (2004)Google Scholar
  4. 4.
    Boykov, Y., Jolly, M.: Interactive graph cuts for optimal boundary region segmentation of objects in n-d images. In: ICCV (2001)Google Scholar
  5. 5.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. on Pattern Analysis and Machine Intelligence 26, 1222–1239 (2001)CrossRefGoogle Scholar
  6. 6.
    Rother, C., Kolmogorov, V., Blake, A.: “grabcut”: Interactive foreground extraction using iterated graph cuts. ACM Trans. on Graphics 23, 309–314 (2004)CrossRefGoogle Scholar
  7. 7.
    Galun, M., Sharon, E., Basri, R., Br, A.: Texture segmentation by multiscale aggregation of filter responses and shape elements. In: ICCV (2003)Google Scholar
  8. 8.
    Riklin-raviv, T., Kiryati, N., Sochen, N.: Segmentation by level sets and symmetry. In: CVPR (2006)Google Scholar
  9. 9.
    Bagon, S., Boiman, O., Irani, M.: What is a good image segment? A unified approach to segment extraction. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part IV. LNCS, vol. 5305, pp. 30–44. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  10. 10.
    Rhemann, C., Rother, C., Wang, J., Gelautz, M., Kohli, P., Rott, P.: A perceptually motivated online benchmark for image matting. In: CVPR (2009)Google Scholar
  11. 11.
    Bai, X., Wang, J., Simons, D., Sapiro, G.: Video snapcut: Robust video object cutout using localized classifiers. In: SIGGRAPH (2009)Google Scholar
  12. 12.
    Li, Y., Sun, J., Shum, H.: Video object cut and paste. ACM Trans. on Graphics 24, 595–600 (2005)CrossRefGoogle Scholar
  13. 13.
    Zhu, J., Liao, M., Yang, R., Pan, Z.: Joint depth and alpha matte optimization via fusion of stereo and time-of-flight sensor. In: CVPR (2009)Google Scholar
  14. 14.
    Xiao, J., Shah, M.: Accurate motion layer segmentation and matting. In: CVPR (2005)Google Scholar
  15. 15.
    Zhang, G., Dong, Z., Jia, J., Wan, L., Wong, T., Bao, H.: Refilming with depth-inferred videos. IEEE Trans. on Visualization and Computer Graphics 15, 828–840 (2009)CrossRefGoogle Scholar
  16. 16.
    Sun, J., Jia, J., Tang, C., Shum, H.: Poisson matting. ACM Trans. on Graphics 23, 315–321 (2004)CrossRefGoogle Scholar
  17. 17.
    Chuang, Y., Agarwala, A., Curless, B., Salesin, D., Szeliski, R.: Video matting of complex scenes. ACM Trans. on Graphics 21, 243–248 (2002)CrossRefGoogle Scholar
  18. 18.
    Lhuillier, M., Quan, L.: Match propagation for image-based modeling and rendering. IEEE Trans. on Pattern Analysis and Machine Intelligence 24, 1140–1146 (2002)CrossRefGoogle Scholar
  19. 19.
    Kolmogorov, V., Zabih, R.: Computing visual correspondence with occlusions via graph cuts. In: ICCV, pp. 508–515 (2001)Google Scholar
  20. 20.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation. Int. J. of Computer Vision 70, 109–131 (2004)Google Scholar
  21. 21.
    Li, Y., Sun, J., Tang, C., Shum, H.: Lazy snapping. ACM Trans. on Graphics 23, 303–308 (2004)CrossRefGoogle Scholar
  22. 22.
    Chuang, Y.Y., Curless, B., Salesin, D.H., Szeliski, R.: A bayesian approach to digital matting. In: CVPR (2001)Google Scholar
  23. 23.
    Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. of Computer Vision 47, 7–42 (2002)CrossRefzbMATHGoogle Scholar
  24. 24.
    Khan, S., Shah, M.: Object based segmentation of video using color, motion and spatial information. In: CVPR (2001)Google Scholar
  25. 25.
    Xiao, J., Shah, M.: Motion layer extraction in the presence of occlusion using graph cut. In: CVPR (2004)Google Scholar
  26. 26.
    Dupont, R., Paragios, N., Keriven, R., Fuchs, P.: Extraction of layers of similar motion through combinatorial techniques. In: Rangarajan, A., Vemuri, B.C., Yuille, A.L. (eds.) EMMCVPR 2005. LNCS, vol. 3757, pp. 220–234. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  27. 27.
    Schoenemann, T., Cremers, D.: High resolution motion layer decomposition using dual-space graph cuts. In: CVPR (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Yanli Li
    • 1
    • 2
  • Zhong Zhou
    • 1
    • 2
  • Wei Wu
    • 1
    • 2
  1. 1.State Key Lab. of Virtual Reality Technology and SystemsBeihang UniversityChina
  2. 2.School of Computer Science and EngineeringBeihang UniversityChina

Personalised recommendations