Multimedia Tools and Applications

, Volume 78, Issue 2, pp 2551–2569 | Cite as

Labeling of partially occluded regions via the multi-layer CRF

  • Sergey KosovEmail author
  • Kimiaki Shirahama
  • Marcin Grzegorzek


This work proposes a general multi-layer framework for image labeling, which targets the challenging problem of classifying the occluded parts of the 3D scene depicted in a 2D image. Our framework is based on the mixed graphical models, which explicitly encode causal relationship between the visible and occluded regions. Unlike other image labeling techniques where a single label is determined for each pixel, layered model assigns multiple labels to pixels. We propose a novel “Multi-Layer-CRF” framework that allows for the integration of sophisticated occlusion potentials into the model and enables the automatic inference of the layer decomposition. We use a special message-passing algorithm to perform maximum a posterior inference on mixed graphs and demonstrate the ability to infer the correct labels of occluded regions in both the aerial near-vertical dataset and urban street-view dataset. It is shown to increase the classification accuracy in occluded areas significantly.


Conditional random fields Graphical models Classification Semantic segmentation Occlusions 



  1. 1.
    Bileschi SM (2006) Streetscenes: towards scene understanding in still images. Ph.D. thesis MITGoogle Scholar
  2. 2.
    Boykov Y, Jolly MP (2001) Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images. In: Proceedings of ICCV, pp 105–112Google Scholar
  3. 3.
    Breiman L. (2001) Random forests. Mach LearnGoogle Scholar
  4. 4.
    Chen YT, Liu X, Yang MH (2015) Multi-instance object segmentation with occlusion handling. In: Proceedings of CVPR, pp 3470–3478Google Scholar
  5. 5.
    Cramer M (2010) The DGPF test on digital aerial camera evaluation - overview and test design. PFG 2(2010):73–82Google Scholar
  6. 6.
    Criminisi A, Shotton J (2013) Decision forests in computer vision and medical image analysis. Springer, BerlinCrossRefGoogle Scholar
  7. 7.
    Duda R, Hart P (1972) Use of the hough transformation to detect lines and curves in pictures. Commun ACM 15(1):11–15CrossRefGoogle Scholar
  8. 8.
    Felzenszwalb PF, Girshick RB, McAllester D, Ramanan D (2010) Object detection with discriminatively trained part-based models. IEEE Trans Pattern Anal Mach Intell 32(9):1627–1645CrossRefGoogle Scholar
  9. 9.
    Frydenberg M (1990) The chain graph Markov property. Scand J Stat 17:333–353MathSciNetzbMATHGoogle Scholar
  10. 10.
    Guo R, Hoiem D (2012) Beyond the line of sight: labeling the underlying surfaces. In: Proceedings of ECCV, pp 761–774Google Scholar
  11. 11.
    Guo R, Hoiem D (2015) Labeling complete surfaces in scene understanding. IJCV 112(2):172–187MathSciNetCrossRefGoogle Scholar
  12. 12.
    Heitz G, Gal C (2010) Object separation in x-ray image sets. In: Proceedings of CVPR, pp 2093–2100Google Scholar
  13. 13.
    Hinz S, Baumgartner A (2003) Automatic extraction of urban road networks from multi-view aerial imagery. ISPRS J Photogramm Remote Sens 58:83–98CrossRefGoogle Scholar
  14. 14.
    Hoiem D, Efros A, Hebert M (2005) Geometric context from a single image. In: Proceedings of ICCV, pp 654–661Google Scholar
  15. 15.
    Kim BS, Kohli P, Savarese S (2013) 3D scene understanding by voxel-CRF. In: Proceedings of ICCV, pp 1425–1432Google Scholar
  16. 16.
    Kolmogorov V (2006) Convergent tree-reweighted message passing for energy minimization. IEEE Trans Pattern Anal Mach Intell 28(10):1568–1583CrossRefGoogle Scholar
  17. 17.
    Kosov S (2015) Direct graphical models C++ library.
  18. 18.
    Kosov S, Kohli P, Rottensteiner F, Heipke C (2013) A two-layer conditional random field for the classification of partially occluded objects. arXiv:1307.3043 [cs.CV]
  19. 19.
    Kosov S, Rottensteiner F, Heipke C, Leitloff J, Hinz S. (2012) 3D classification of crossroads from multiple aerial images using Markov random fields. In: Proceedings of ISPRS Congress, pp 479–484Google Scholar
  20. 20.
    Kramer O (2010) Iterated local search with Powell’s method: a memetic algorithm for continuous global optimization. Memetic Comput 2(1):69–83CrossRefGoogle Scholar
  21. 21.
    Kulkarni VY, Sinha PK (2013) Random forest classifiers: a survey and future research directions. Int J Adv Comput 36(1):1144–1153Google Scholar
  22. 22.
    Kumar S, Hebert M (2005) A hierarchical field framework for unified context-based classification. In: Proceedings of ICCV, pp 1284–1291Google Scholar
  23. 23.
    Lafferty J, McCallum A, Pereira F (2001) Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of ICML, pp 282–289Google Scholar
  24. 24.
    Lauritzen S, Wermuth N (1989) Graphical models for associations between variables, some of which are qualitative and some quantitative. Ann Stat 17(1):31–57MathSciNetCrossRefGoogle Scholar
  25. 25.
    Leibe B, Leonardis A, Schiele B (2008) Robust object detection with interleaved categorization and segmentation. IJCV 77:259–289CrossRefGoogle Scholar
  26. 26.
    Li SZ (2009) Markov random field modeling in image analysis, 3rd edn. Springer, BerlinzbMATHGoogle Scholar
  27. 27.
    Liu C, Kohli P, Furukawa Y (2016) Layered scene decomposition via the occlusion-CRF. In: Proceedings of CVPR, pp 165–173Google Scholar
  28. 28.
    Liu C, Yuen J, Torralba A (2011) Nonparametric scene parsing via label transfer. IEEE Trans Pattern Anal Mach Intell 33(12):2368–2382CrossRefGoogle Scholar
  29. 29.
    Prasad M, Zisserman A, Fitzgibbon A, Kumar P, Torr P (2006) Learning class-specific edges for object detection and segmentation. In: Proceedings of ICVGIP, pp 94–105Google Scholar
  30. 30.
    Richardson T, Spirtes P (2002) Ancestral graph Markov models. Ann Stat 30 (4):962–1030MathSciNetCrossRefGoogle Scholar
  31. 31.
    Schindler K (2012) An overview and comparison of smooth labeling methods for land-cover classification. IEEE Trans Geosci Remote Sens:4534–4545Google Scholar
  32. 32.
    Schnitzspan P, Fritz M, Roth S, Schiele B (2009) Discriminative structure learning of hierarchical representations for object detection. In: Proceedings of CVPR, pp 2238–2245Google Scholar
  33. 33.
    Silberman N, Shapira L, Gal R, Kohli P (2014) A contour completion model for augmenting surface reconstructions. In: Proceedings of ECCV, pp 488–503Google Scholar
  34. 34.
    Souly N, Shah M (2016) Scene labeling using sparse precision matrix. In: Proceedings of CVPR. IEEE Computer Society, pp 3650–3658Google Scholar
  35. 35.
    Souly N, Spampinato C, Shah M (2017) Semi supervised semantic segmentation using generative adversarial network. In: Proceedings of ICCV, pp 5689–5697Google Scholar
  36. 36.
    Szeliski R, Zabih R, Scharstein D, Veksler O, Kolmogorov V, Agarwala A, Tappen M, Rother C (2008) A comparative study of energy minimization methods for Markov random fields with smoothness-based priors. IEEE Trans Pattern Anal Mach Intell 30(6):1068–1080CrossRefGoogle Scholar
  37. 37.
    Tighe J, Niethammer M, Lazebnik S (2014) Scene parsing with object instances and occlusion ordering. In: Proceedings of CVPR, pp 3748–3755Google Scholar
  38. 38.
    Tu Z, Bai X (2010) Auto-context and its application to high-level vision tasks and 3D brain image segmentation. IEEE Trans Pattern Anal Mach Intell 32(10):1744–1757CrossRefGoogle Scholar
  39. 39.
    Vishwanathan S, Schraudolph N, Schmidt M, Murphy K (2006) Accelerated training of conditional random fields with stochastic gradient methods. In: Proceedings of ICML, pp 969–976Google Scholar
  40. 40.
    Winn J, Shotton J (2006) The layout consistent random field for recognizing and segmenting partially occluded objects. In: Proceedings of CVPR, pp 37–44Google Scholar
  41. 41.
    Yang Y, Hallman S, Ramanan D, Fowlkes C (2012) Layered object models for image segmentation. IEEE Trans Pattern Anal Mach Intell 34:1731–1743CrossRefGoogle Scholar
  42. 42.
    Yin Z, Collins R (2007) Belief propagation in a 3D spatiotemporal MRF for moving object detection. In: Proceedings of CVPR, pp 1–8Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Research Group for Pattern Recognition, Department ETIUniversity of SiegenSiegenGermany

Personalised recommendations