Graph Cut Based Inference with Co-occurrence Statistics

  • Lubor Ladicky
  • Chris Russell
  • Pushmeet Kohli
  • Philip H. S. Torr
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6315)


Markov and Conditional random fields (crfs) used in computer vision typically model only local interactions between variables, as this is computationally tractable. In this paper we consider a class of global potentials defined over all variables in the crf. We show how they can be readily optimised using standard graph cut algorithms at little extra expense compared to a standard pairwise field.

This result can be directly used for the problem of class based image segmentation which has seen increasing recent interest within computer vision. Here the aim is to assign a label to each pixel of a given image from a set of possible object classes. Typically these methods use random fields to model local interactions between pixels or super-pixels. One of the cues that helps recognition is global object co-occurrence statistics, a measure of which classes (such as chair or motorbike) are likely to occur in the same image together. There have been several approaches proposed to exploit this property, but all of them suffer from different limitations and typically carry a high computational cost, preventing their application on large images. We find that the new model we propose produces an improvement in the labelling compared to just using a pairwise model.


Object Class Graph Construction Move Energy Pairwise Potential Swap Move 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Benson, H.Y., Shanno, D.F.: An exact primal—dual penalty method approach to warmstarting interior-point methods for linear programming. Comput. Optim. Appl. (2007)Google Scholar
  2. 2.
    Borenstein, E., Malik, J.: Shape guided object segmentation. In: CVPR (2006)Google Scholar
  3. 3.
    Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. PAMI (2001)Google Scholar
  4. 4.
    Choi, M.J., Lim, J.J., Torralba, A., Willsky, A.S.: Exploiting hierarchical context on a large database of object categories. In: CVPR (2010)Google Scholar
  5. 5.
    Comaniciu, D., Meer, P.: Mean shift: A robust approach toward feature space analysis. PAMI (2002)Google Scholar
  6. 6.
    Csurka, G., Perronnin, F.: A simple high performance approach to semantic segmentation. In: BMVC 2008 (2008)Google Scholar
  7. 7.
    Delong, A., Osokin, A., Isack, H., Boykov, Y.: Fast approximate energy minimization with label costs. In: CVPR (2010)Google Scholar
  8. 8.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient graph-based image segmentation. IJCV (2004)Google Scholar
  9. 9.
    Galleguillos, C., Rabinovich, A., Belongie, S.: Object categorization using co-occurrence, location and appearance. In: CVPR (2008)Google Scholar
  10. 10.
    Gould, S., Fulton, R., Koller, D.: Decomposing a scene into geometric and semantically consistent regions. In: ICCV (2009)Google Scholar
  11. 11.
    Heitz, D.K.G.: Learning spatial context: Using stuff to find things. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008, Part I. LNCS, vol. 5302, pp. 30–43. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  12. 12.
    Hoiem, D., Rother, C., Winn, J.M.: 3d layoutcrf for multi-view object class recognition and segmentation. In: CVPR (2007)Google Scholar
  13. 13.
    Kohli, P., Ladicky, L., Torr, P.H.: Robust higher order potentials for enforcing label consistency. In: CVPR (2008)Google Scholar
  14. 14.
    Kolmogorov, V.: Convergent tree-reweighted message passing for energy minimization. PAMI (2006)Google Scholar
  15. 15.
    Kolmogorov, V., Rother, C.: Comparison of energy minimization algorithms for highly connected graphs. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 1–15. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  16. 16.
    Ladicky, L., Russell, C., Kohli, P., Torr, P.H.: Graph Cut based Inference with Co-occurrence Statistics — Technical report (2010)Google Scholar
  17. 17.
    Ladicky, L., Russell, C., Kohli, P., Torr, P.H.: Associative hierarchical crfs for object class image segmentation. In: ICCV (2009)Google Scholar
  18. 18.
    Ladicky, L., Russell, C., Sturgess, P., Alahari, K., Torr, P.H.: What, where and how many? Combining object detectors and CRFs. In: ECCV (2010)Google Scholar
  19. 19.
    Lafferty, J., McCallum, A., Pereira, F.: Conditional random fields: Probabilistic models for segmenting and labelling sequence data. In: ICML (2001)Google Scholar
  20. 20.
    Larlus, D., Jurie, F.: Combining appearance models and markov random fields for category level object segmentation. In: CVPR (2008)Google Scholar
  21. 21.
    Narasimhan, M., Bilmes, J.A.: A submodular-supermodular procedure with applications to discriminative structure learning. In: UAI (2005)Google Scholar
  22. 22.
    Rabinovich, A., Vedaldi, A., Galleguillos, C., Wiewiora, E., Belongie, S.: Objects in context. In: ICCV (2007)Google Scholar
  23. 23.
    Ren, X., Fowlkes, C., Malik, J.: Mid-level cues improve boundary detection. Technical Report UCB/CSD-05-1382, Berkeley (March 2005)Google Scholar
  24. 24.
    Rother, C., Kumar, S., Kolmogorov, V., Blake, A.: Digital tapestry. In: CVPR (2005)Google Scholar
  25. 25.
    Russell, B., Freeman, W., Efros, A., Sivic, J., Zisserman, A.: Using multiple segmentations to discover objects and their extent in image collections. In: CVPR (2006)Google Scholar
  26. 26.
    Russell, C., Ladicky, L., Kohli, P., Torr, P.H.: Exact and approximate inference in associative hierarchical networks using graph cuts. In: UAI (2010)Google Scholar
  27. 27.
    Schölkopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. In: Adaptive Computation and Machine Learning. MIT Press, Cambridge (2001)Google Scholar
  28. 28.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. PAMI (2000)Google Scholar
  29. 29.
    Shotton, J., Winn, J., Rother, C., Criminisi, A.: TextonBoost: Joint appearance, shape and context modeling for multi-class object recognition and segmentation. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006, Part I. LNCS, vol. 3951, pp. 1–15. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  30. 30.
    Szeliski, R., Zabih, R., Scharstein, D., Veksler, O., Kolmogorov, V., Agarwala, A., Tappen, M., Rother, C.: A comparative study of energy minimization methods for markov random fields. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 16–29. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  31. 31.
    Torralba, A., Murphy, K.P., Freeman, W.T., Rubin, M.A.: Context-based vision system for place and object recognition. In: Proceedings of Computer Vision (2003)Google Scholar
  32. 32.
    Toyoda, T., Hasegawa, O.: Random field model for integration of local information and global information. PAMI (2008)Google Scholar
  33. 33.
    Weiss, Y., Freeman, W.: On the optimality of solutions of the max-product belief-propagation algorithm in arbitrary graphs. Transactions on Information Theory (2001)Google Scholar
  34. 34.
    Yang, L., Meer, P., Foran, D.J.: Multiple class segmentation using a unified framework over mean-shift patches. In: CVPR (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Lubor Ladicky
    • 1
  • Chris Russell
    • 1
  • Pushmeet Kohli
    • 2
  • Philip H. S. Torr
    • 1
  1. 1.Oxford Brookes 
  2. 2.Microsoft Research 

Personalised recommendations