A Dynamic Conditional Random Field Model for Joint Labeling of Object and Scene Classes

  • Christian Wojek
  • Bernt Schiele
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5305)


Object detection and pixel-wise scene labeling have both been active research areas in recent years and impressive results have been reported for both tasks separately. The integration of these different types of approaches should boost performance for both tasks as object detection can profit from powerful scene labeling and also pixel-wise scene labeling can profit from powerful object detection. Consequently, first approaches have been proposed that aim to integrate both object detection and scene labeling in one framework. This paper proposes a novel approach based on conditional random field (CRF) models that extends existing work by 1) formulating the integration as a joint labeling problem of object and scene classes and 2) by systematically integrating dynamic information for the object detection task as well as for the scene labeling task. As a result, the approach is applicable to highly dynamic scenes including both fast camera and object movements. Experiments show the applicability of the novel approach to challenging real-world video sequences and systematically analyze the contribution of different system components to the overall performance.


Input Image Object Detection Object Class Conditional Random Field Dynamic Scene 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Supplementary material

978-3-540-88693-8_54_MOESM1_ESM.avi (1.2 mb)
Supplementary material(1,196 KB)


  1. 1.
    Everingham, M., Zisserman, A., Williams, C., van Gool, L.: The pascal visual object classes challenge results. Technical report, PASCAL Network (2006)Google Scholar
  2. 2.
    Torralba, A.: Contextual priming for object detection. IJCV, 169–191 (2003)Google Scholar
  3. 3.
    Hoiem, D., Efros, A.A., Hebert, M.: Putting objects in perspective. In: CVPR (2006)Google Scholar
  4. 4.
    He, X., Zemel, R.S., Carreira-Perpiñán, M.Á.: Multiscale conditional random fields for image labeling. In: CVPR (2004)Google Scholar
  5. 5.
    Kumar, S., Hebert, M.: A hierarchical field framework for unified context-based classification. In: ICCV (2005)Google Scholar
  6. 6.
    Shotton, J., Winn, J., Rother, C., Criminisi, A.: Textonboost: Joint appearance, shape and context modeling for multi-class object recognition and segmentation. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  7. 7.
    Torralba, A., Murphy, K.P., Freeman, W.T.: Contextual models for object detection using boosted random fields. In: NIPS (2004)Google Scholar
  8. 8.
    McCallum, A., Rohanimanesh, K., Sutton, C.: Dynamic conditional random fields for jointly labeling multiple sequences. In: NIPS Workshop on Syntax, Semantics (2003)Google Scholar
  9. 9.
    Lafferty, J.D., McCallum, A., Pereira, F.C.N.: Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In: ICML (2001)Google Scholar
  10. 10.
    Kumar, S., Hebert, M.: Discriminative random fields: A discriminative framework for contextual interaction in classification. In: ICCV (2003)Google Scholar
  11. 11.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR (2005)Google Scholar
  12. 12.
    Verbeek, J., Triggs, B.: Region classification with markov field aspect models. In: CVPR (2007)Google Scholar
  13. 13.
    Quattoni, A., Collins, M., Darrell, T.: Conditional random fields for object recognition. In: NIPS (2004)Google Scholar
  14. 14.
    Kapoor, A., Winn, J.: Located hidden random fields: Learning discriminative parts for object detection. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3954. Springer, Heidelberg (2006)Google Scholar
  15. 15.
    Winn, J., Shotton, J.: The layout consistent random field for recognizing and segmenting partially occluded objects. In: CVPR (2006)Google Scholar
  16. 16.
    Wang, Y., Ji, Q.: A dynamic conditional random field model for object segmentation in image sequences. In: CVPR (2005)Google Scholar
  17. 17.
    Yin, Z., Collins, R.: Belief propagation in a 3D spatio-temporal MRF for moving object detection. In: CVPR (2007)Google Scholar
  18. 18.
    Leibe, B., Cornelis, N., Cornelis, K., Van Gool, L.: Dynamic 3D scene analysis from a moving vehicle. In: CVPR (2007)Google Scholar
  19. 19.
    Torralba, A., Murphy, K.P., Freeman, W.T.: Sharing features: Efficient boosting procedures for multiclass object detection. In: CVPR (2004)Google Scholar
  20. 20.
    Platt, J.: Probabilistic outputs for support vector machines and comparison to regularized likelihood methods. In: Smola, A.J., Bartlett, P., Schoelkopf, B., Schuurmans, D. (eds.) Advances in Large Margin Classifiers, pp. 61–74 (2000)Google Scholar
  21. 21.
    Kalman, R.E.: A new approach to linear filtering and prediction problems. Transactions of the ASME–Journal of Basic Engineering 82, 35–45 (1960)Google Scholar
  22. 22.
    Sutton, C., McCallum, A.: Piecewise training for undirected models. In: 21th Annual Conference on Uncertainty in Artificial Intelligence (UAI 2005) (2005)Google Scholar
  23. 23.
    Vishwanathan, S.V.N., Schraudolph, N.N., Schmidt, M.W., Murphy, K.P.: Accelerated training of conditional random fields with stochastic gradient methods. In: ICML (2006)Google Scholar
  24. 24.
    Hel-Or, Y., Hel-Or, H.: Real-time pattern matching using projection kernels. PAMI 27, 1430–1445 (2005)Google Scholar
  25. 25.
    Alon, Y., Ferencz, A., Shashua, A.: Off-road path following using region classification and geometric projection constraints. In: CVPR (2006)Google Scholar
  26. 26.
    Varma, M., Zisserman, A.: Classifying images of materials: Achieving viewpoint and illumination independence. In: Tistarelli, M., Bigun, J., Jain, A.K. (eds.) ECCV 2002. LNCS, vol. 2359. Springer, Heidelberg (2002)Google Scholar
  27. 27.
    Russell, B.C., Torralba, A., Murphy, K.P., Freeman, W.T.: LabelMe: A database and web-based tool for image annotation. IJCV 77, 157–173 (2008)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Christian Wojek
    • 1
  • Bernt Schiele
    • 1
  1. 1.Computer Science DepartmentTU Darmstadt 

Personalised recommendations