Time-Consistent Foreground Segmentation of Dynamic Content from Color and Depth Video

  • Anatol Frick
  • Markus Franke
  • Reinhard Koch
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6835)


This paper introduces an approach for automatic foreground extraction from videos utilizing depth information from time of flight(ToF) cameras. We give a clear definition of background and foreground based on 3D scene geometry and provide means of foreground extraction based on one-dimensional histograms in 3D space. Further a refinement step based on hierarchical grab-cut segmentation in a video volume with incorporated time constraints is proposed. Our approach is able to extract time-consistent foreground objects even for a moving camera and for dynamic scene content, but is limited to indoor scenarios.


Gaussian Mixture Model Depth Image Foreground Object Uncertainty Region Thresholded Image 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Bartczak, B., Schiller, I., Beder, C., Koch, R.: Integration of a time-of-flight camera into a mixed reality system for handling dynamic scenes, moving viewpoints and occlusions in real-time. In: Proceedings of the 3DPVT Workshop (2008)Google Scholar
  2. 2.
    Boykov, Y., Funka-Lea, G.: Graph cuts and efficient nd image segmentation. International Journal of Computer Vision 70(2), 109–131 (2006)CrossRefGoogle Scholar
  3. 3.
    Boykov, Y., Kolmogorov, V.: An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision. IEEE Transactions on Pattern Analysis and Machine Intelligence 26(9), 1124–1137 (2004)CrossRefGoogle Scholar
  4. 4.
    Crabb, R., Tracey, C., Puranik, A., Davis, J.: Real-time foreground segmentation via range and color imaging. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, CVPRW 2008, pp. 1–5 (2008)Google Scholar
  5. 5.
    Criminisi, A., Cross, G., Blake, A., Kolmogorov, V.: Bilayer Segmentation of Live Video. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 53–60 (2006)Google Scholar
  6. 6.
    Fischler, M., Bolles, R.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM 24(6), 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  7. 7.
    Frick, A., Bartczak, B., Koch, R.: Real-time preview for layered depth video in 3D-TV. In: Proceedings of SPIE 7724, 77240F (2010)Google Scholar
  8. 8.
    Kolb, A., Barth, E., Koch, R., Larsen, R.: Time-of-flight sensors in computer graphic. In: Eurographics 2009 - State of the Art Reports pp. 119–134 (2009)Google Scholar
  9. 9.
    Kolmogorov, V., Criminisi, A., Blake, A., Cross, G., Rother, C.: Bi-layer segmentation of binocular stereo video. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2005, vol. 2, pp. 407–414 (2005)Google Scholar
  10. 10.
    Li, Y., Sun, J., Shum, H.: Video object cut and paste. ACM Transactions on Graphics (TOG) 24(3), 600 (2005)Google Scholar
  11. 11.
    Li, Y., Sun, J., Tang, C., Shum, H.: Lazy snapping. ACM Transactions on Graphics (TOG) 23(3), 303–308 (2004)CrossRefGoogle Scholar
  12. 12.
    Orchard, M., Bouman, C.: Color quantization of images. IEEE Transactions on Signal Processing 39(12), 2677–2690 (1991)CrossRefGoogle Scholar
  13. 13.
    Pham, V., Takahashi, K., Naemura, T.: Live video segmentation in dynamic backgrounds using thermal vision. Advances in Image and Video Technology, 143–154 (2009)Google Scholar
  14. 14.
    Rother, C., Kolmogorov, V., Blake, A.: Grabcut: Interactive foreground extraction using iterated graph cuts. ACM Transactions on Graphics 23, 309–314 (2004)CrossRefGoogle Scholar
  15. 15.
    Schiller, I., Beder, C., Koch, R.: Calibration of A PMD-Camera using a Planar Calibration Pattern Together with a Multi-camera Setup. In: Proc. XXXVII Int’l Soc. for Photogrammetry (2008)Google Scholar
  16. 16.
    Smith, A., Blinn, J.: Blue screen matting. In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 259–268 (1996)Google Scholar
  17. 17.
    Sun, J., Zhang, W., Tang, X., Shum, H.-Y.: Background cut. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 628–641. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  18. 18.
    Wang, O., Finger, J., Yang, Q., Davis, J., Yang, R.: Automatic Natural Video Matting with Depth. In: 15th Pacific Conference on Computer Graphics and Applications, PG 2007, pp. 469–472 (2007)Google Scholar
  19. 19.
    Wu, Q., Boulanger, P., Bischof, W.: Robust Real-Time Bi-Layer Video Segmentation Using Infrared Video. In: Canadian Conference on Computer and Robot Vision, CRV 2008, pp. 87–94 (2008)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Anatol Frick
    • 1
  • Markus Franke
    • 1
  • Reinhard Koch
    • 1
  1. 1.Computer Science DepartmentChristian-Albrechts-University KielKielGermany

Personalised recommendations