Improved Video Segmentation by Adaptive Combination of Depth Keying and Mixture-of-Gaussians

  • Ingo Schiller
  • Reinhard Koch
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6688)


Video segmentation or matting, the separation of foreground objects from background in video sequences, is a demanding task and is needed for a broad range of applications. The most widespread method for video segmentation is chroma-keying using a known background color for which a controlled environment is required. Recently a different method of keying fore-and background has been proposed in which the chroma-keying is replaced by depth-keying using a Time-of-Flight (ToF) camera. The current ToF-cameras suffer from noise and low resolution sensors, which results in unsatisfying segmentation results. We propose to combine the segmentation of dynamic objects in depth with a segmentation in the color domain using adaptive background models. We weight the two measures depending on the actual depth values using either the variance of the depth images of the ToF-camera or the amplitude image of the ToF-camera as reliability measure. We show that both methods significantly improve the segmentation results.


Segmentation Result Depth Image Foreground Object Color Segmentation Amplitude Image 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Bartczak, B., Schiller, I., Beder, C., Koch, R.: Integration of a time-of-flight camera into a mixed reality system for handling dynamic scenes, moving viewpoints and occlusions in real-time. In: Proceedings of the 3DPVT Workshop, Atlanta, GA, USA (June 2008)Google Scholar
  2. 2.
    Crabb, R., Tracey, C., Puranik, A., Davis, J.: Real-time foreground segmentation via range and color imaging. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1–5 (June 2008)Google Scholar
  3. 3.
    Gong, M., Wang, L., Yang, R., Yang, Y.H.: Real-time video matting using multichannel poisson equations. In: Proceedings of Graphics Interface (GI), pp. 89–96. Canadian Information Processing Society, Toronto (2010),
  4. 4.
    Gordon, G., Darrell, T., Harville, M., Woodfill, J.: Background estimation and removal based on range and color. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), vol. 2, pp. 458–464 (1999)Google Scholar
  5. 5.
    Gvili, R., Kaplan, A., Ofek, E., Yahav, G.: Depth keying, vol. 5006, pp. 564–574. SPIE (2003),
  6. 6.
    Smith, A.R., Blinn, J.F.: Blue screen matting. In: SIGGRAPH 1996: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, pp. 259–268. ACM, New York (1996)CrossRefGoogle Scholar
  7. 7.
    Stauffer, C., Eric, W., Grimson, L.: Adaptive background mixture models for real-time tracking. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2246–2252 (1999)Google Scholar
  8. 8.
    Wang, J., Cohen, M.F.: Image and video matting: a survey. Foundations and Trends in Computer Graphics and Vision 3(2), 97–175 (2007)CrossRefGoogle Scholar
  9. 9.
    Xu, M., Ellis, T.: Illumination-invariant motion detection using colour mixture models. In: British Machine Vision Conference (BMVC), pp. 163–172 (2001)Google Scholar
  10. 10.
    Xu, Z., Schwarte, R., Heinol, H., Buxbaum, B., Ringbeck, T.: Smart pixel - photonic mixer device (PMD). In: International Conference on Mechatronics and Machine Vision in Practice (M2VIP), Nanjing, p. 259–264 (1998)Google Scholar
  11. 11.
    Zhu, J., Liao, M., Yang, R., Pan, Z.: Joint depth and alpha matte optimization via fusion of stereo and time-of-flight sensor. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp. 453–460. IEEE, Los Alamitos (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Ingo Schiller
    • 1
  • Reinhard Koch
    • 1
  1. 1.Multimedia Information Processing (MIP) Institute of Computer ScienceUniversity of KielGermany

Personalised recommendations