Interactive Multi-label Segmentation of RGB-D Images

  • Julia DieboldEmail author
  • Nikolaus Demmel
  • Caner Hazırbaş
  • Michael Moeller
  • Daniel Cremers
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9087)


We propose a novel interactive multi-label RGB-D image segmentation method by extending spatially varying color distributions [14] to additionally utilize depth information in two different ways. On the one hand, we consider the depth image as an additional data channel. On the other hand, we extend the idea of spatially varying color distributions in a plane to volumetrically varying color distributions in 3D. Furthermore, we improve the data fidelity term by locally adapting the influence of nearby scribbles around each pixel. Our approach is implemented for parallel hardware and evaluated on a novel interactive RGB-D image segmentation benchmark with pixel-accurate ground truth. We show that depth information leads to considerably more precise segmentation results. At the same time significantly less user scribbles are required for obtaining the same segmentation accuracy as without using depth clues.


Multi-label segmentation RGB-D images Interactive segmentation Spatially varying color distributions Total variation 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Arbelaez, P., Maire, M., Fowlkes, C.C., Malik, J.: From contours to regions: an empirical evaluation. In: CVPR (2009)Google Scholar
  2. 2.
    Batra, D., Kowdle, A., Parikh, D., Luo, J., Chen, T.: iCoseg: interactive co-segmentation with intelligent scribble guidance. In: CVPR (2010)Google Scholar
  3. 3.
    Blake, A., Rother, C., Brown, M., Perez, P., Torr, P.: Interactive image segmentation using an adaptive GMMRF model. In: Pajdla, T., Matas, J.G. (eds.) ECCV 2004. LNCS, vol. 3021, pp. 428–441. Springer, Heidelberg (2004) Google Scholar
  4. 4.
    Boykov, Y.Y., Jolly, M.-P.: Interactive graph cuts for optimal boundary & region segmentation of objects in ND images. In: ICCV (2001)Google Scholar
  5. 5.
    Couprie, C., Farabet, C., Najman, L., LeCun, Y.: Indoor semantic segmentation using depth information. In: ICLR (2013)Google Scholar
  6. 6.
    Esser, E., Zhang, X., Chan, T.F.: A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science. SIIMS (2010)Google Scholar
  7. 7.
    Hermans, A., Floros, G., Leibe, B.: Dense 3D semantic mapping of indoor scenes from RGB-D images. In: ICRA (2014)Google Scholar
  8. 8.
    Hernandez, J. Marcotegui, B.: Morphological segmentation of building facade images. In: ICIP (2009)Google Scholar
  9. 9.
    Kohli, P., Ladicky, L., Torr, P.H.S.: Robust higher order potentials for enforcing label consistency. IJCV (2009)Google Scholar
  10. 10.
    Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. In: TOG (2004)Google Scholar
  11. 11.
    Li, Y., Sun, J., Tang, C.-K., Shum, H.-Y.: Lazy snapping. TOG (2004)Google Scholar
  12. 12.
    Liu, D., Pulli, K., Shapiro, L.G., Xiong, Y.: Fast interactive image segmentation by discriminative clustering. In: MCMC (2010)Google Scholar
  13. 13.
    Lombaert, H., Sun, Y., Grady, L., Xu, C.: A multilevel banded graph cuts method for fast image segmentation. In: ICCV (2005)Google Scholar
  14. 14.
    Nieuwenhuis, C., Cremers, D.: Spatially varying color distributions for interactive multilabel segmentation. PAMI (2013)Google Scholar
  15. 15.
    Nieuwenhuis, C., Hawe, S., Kleinsteuber, M., Cremers, D.: Co-sparse textural similarity for interactive segmentation. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part VI. LNCS, vol. 8694, pp. 285–301. Springer, Heidelberg (2014) Google Scholar
  16. 16.
    Pock, T., Chambolle, A.: Diagonal preconditioning for first order primal-dual algorithms in convex optimization. In: ICCV (2011)Google Scholar
  17. 17.
    Pock, T., Cremers, D., Bischof, H., Chambolle, A.: An algorithm for minimizing the mumford-shah functional. In: ICCV (2009)Google Scholar
  18. 18.
    Richtsfeld, A., Morwald, T., Prankl, J., Zillich, M., Vincze, M.: Segmentation of unknown objects in indoor environments. In: IROS (2012)Google Scholar
  19. 19.
    Rosman, G., Bronstein, A.M., Bronstein, M.M., Tai, X.-C., Kimmel, R.: Group-valued regularization for analysis of articulated motion. In: Fusiello, A., Murino, V., Cucchiara, R. (eds.) ECCV 2012 Ws/Demos, Part I. LNCS, vol. 7583, pp. 52–62. Springer, Heidelberg (2012) Google Scholar
  20. 20.
    Rother, C., Kolmogorov, V., Blake, A.: Grabcut: interactive foreground extraction using iterated graph cuts. In: TOG (2004)Google Scholar
  21. 21.
    Santner, J., Pock, T., Bischof, H.: Interactive multi-label segmentation. In: Kimmel, R., Klette, R., Sugimoto, A. (eds.) ACCV 2010, Part I. LNCS, vol. 6492, pp. 397–410. Springer, Heidelberg (2011) Google Scholar
  22. 22.
    Shao, T., Xu, W., Zhou, K., Wang, J., Li, D., Guo, B.: An interactive approach to semantic modeling of indoor scenes with an rgbd camera. TOG (2012)Google Scholar
  23. 23.
    Silberman, N., Fergus, R.: Indoor scene segmentation using a structured light sensor. In: ICCV (2011)Google Scholar
  24. 24.
    Silberman, N., Hoiem, D., Kohli, P., Fergus, R.: Indoor segmentation and support inference from RGBD images. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012, Part V. LNCS, vol. 7576, pp. 746–760. Springer, Heidelberg (2012) Google Scholar
  25. 25.
    Silverman, B.: Density estimation for statistics and data analysis. Chapman and Hall Ltd (1986)Google Scholar
  26. 26.
    Teboul, O., Simon, L., Koutsourakis, P., Paragios, N.: Segmentation of building facades using procedural shape priors. In: CVPR (2010)Google Scholar
  27. 27.
    Vicente, S., Kolmogorov, V., Rother, C.: Joint optimization of segmentation and appearance models. In: ICCV (2009)Google Scholar
  28. 28.
    Wang, J.: Discriminative gaussian mixtures for interactive image segmentation. In: ICASSP (2007)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Julia Diebold
    • 1
    Email author
  • Nikolaus Demmel
    • 1
  • Caner Hazırbaş
    • 1
  • Michael Moeller
    • 1
  • Daniel Cremers
    • 1
  1. 1.Technical University of MunichMünchenGermany

Personalised recommendations