Pottics – The Potts Topic Model for Semantic Image Segmentation

  • Christoph Dann
  • Peter Gehler
  • Stefan Roth
  • Sebastian Nowozin
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7476)


We present a novel conditional random field (CRF) for semantic segmentation that extends the common Potts model of spatial coherency with latent topics, which capture higher-order spatial relations of segment labels. Specifically, we show how recent approaches for producing sets of figure-ground segmentations can be leveraged to construct a suitable graph representation for this task. The CRF model incorporates such proposal segmentations as topics, modelling the joint occurrence or absence of object classes. The resulting model is trained using a structured large margin approach with latent variables. Experimental results on the challenging VOC’10 dataset demonstrate significant performance improvements over simpler models with less spatial structure.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Alexe, B., Deselaers, T., Ferrari, V.: What is an object? In: CVPR (2010)Google Scholar
  2. 2.
    Batra, D., Sukthankar, R., Chen, T.: Learning class-specific affinities for image labelling. In: CVPR (2008)Google Scholar
  3. 3.
    Carreira, J., Li, F., Sminchisescu, C.: Object Recognition by Sequential Figure-Ground Ranking. IJCV (August 2011)Google Scholar
  4. 4.
    Carreira, J., Ion, A., Sminchisescu, C.: Image segmentation by discounted cumulative ranking on maximal cliques. arXiv CoRR abs/1009.4823 (2010)Google Scholar
  5. 5.
    Carreira, J., Sminchisescu, C.: Constrained parametric min-cuts for automatic object segmentation. In: CVPR, pp. 3241–3248 (2010)Google Scholar
  6. 6.
    Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The Pascal Visual Object Classes (VOC) Challenge. IJCV 88(2), 303–338 (2009)CrossRefGoogle Scholar
  7. 7.
    Fulkerson, B., Vedaldi, A., Soatto, S.: Class segmentation and object localization with superpixel neighborhoods. In: ICCV (2009)Google Scholar
  8. 8.
    Gonfaus, J., Boix, X., van de Weijer, J., Bagdanov, A., Serrat, J., Gonzalez, J.: Harmony potentials for joint classification and segmentation. In: CVPR (2010)Google Scholar
  9. 9.
    Ion, A., Carreira, J., Sminchisescu, C.: Image segmentation by figure-ground composition into maximal cliques. In: ICCV (2011)Google Scholar
  10. 10.
    Ion, A., Carreira, J., Sminchisescu, C.: Probabilistic joint image segmentation and labeling. In: NIPS (2011)Google Scholar
  11. 11.
    Kolmogorov, V.: Convergent tree-reweighted message passing for energy minimization. PAMI 28(10), 1568–1583 (2006)CrossRefGoogle Scholar
  12. 12.
    Krähenbühl, P., Koltun, V.: Efficient inference in fully connected crfs with gaussian edge potentials. In: NIPS (2011)Google Scholar
  13. 13.
    Ladicky, L., Russell, C., Kohli, P., Torr, P.H.S.: Associative hierarchical CRFs for object class image segmentation. In: ICCV. IEEE (September 2009)Google Scholar
  14. 14.
    Nowozin, S., Gehler, P.V., Lampert, C.H.: On Parameter Learning in CRF-Based Approaches to Object Class Image Segmentation. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part VI. LNCS, vol. 6316, pp. 98–111. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  15. 15.
    Nowozin, S., Lampert, C.: Structured Learning and Prediction in Computer Vision. Foundations and Trends in Computer Graphics and Vision 6(3-4), 185–365 (2011)zbMATHCrossRefGoogle Scholar
  16. 16.
    Pantofaru, C., Schmid, C., Hebert, M.: Object Recognition by Integrating Multiple Image Segmentations. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part III. LNCS, vol. 5304, pp. 481–494. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  17. 17.
    Ren, X., Malik, J., Division, C.S.: Learning a classification model for segmentation. In: ICCV (2003)Google Scholar
  18. 18.
    Shotton, J., Winn, J., Rother, C., Criminisi, A.: Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context. IJCV (2007)Google Scholar
  19. 19.
    Tarlow, D., Zemel, R.: Big and tall: Large margin learning with high order losses. In: CVPR Ws. on Inference in Graphical Models with Structured Potentials (2011)Google Scholar
  20. 20.
    Vishwanathan, S.V.N., Schraudolph, N.N., Schmidt, M.W., Murphy, K.P.: Accelerated training of conditional random fields with stochastic gradient methods. In: ICML (2006)Google Scholar
  21. 21.
    Yu, C.N., Joachims, T.: Learning structural svms with latent variables. In: ICML (2009)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Christoph Dann
    • 1
  • Peter Gehler
    • 2
  • Stefan Roth
    • 1
  • Sebastian Nowozin
    • 3
  1. 1.Technische Universität DarmstadtGermany
  2. 2.Max Planck Institute for Intelligent SystemsGermany
  3. 3.Microsoft ResearchCambridgeUK

Personalised recommendations