Kernel Conditional Ordinal Random Fields for Temporal Segmentation of Facial Action Units

  • Ognjen Rudovic
  • Vladimir Pavlovic
  • Maja Pantic
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7584)


We consider the problem of automated recognition of temporal segments (neutral, onset, apex and offset) of Facial Action Units. To this end, we propose the Laplacian-regularized Kernel Conditional Ordinal Random Field model. In contrast to standard modeling approaches to recognition of AUs’ temporal segments, which treat each segment as an independent class, the proposed model takes into account ordinal relations between the segments. The experimental results evidence the effectiveness of such an approach.


Action units histogram intersection kernel ordinal regression conditional random field kernel locality preserving projections 


  1. 1.
    Pantic, M.: Machine analysis of facial behaviour: Naturalistic and dynamic behaviour. Philosophical Transactions of Royal Society B 364, 3505–3513 (2009)CrossRefGoogle Scholar
  2. 2.
    Ekman, P., Friesen, W., Hager, J.: Facial Action Coding System (FACS): Manual. A Human Face (2002)Google Scholar
  3. 3.
    Mahoor, M., Cadavid, S., Messinger, D., Cohn, J.: A framework for automated measurement of the intensity of non-posed facial action units. In: CVPRW, pp. 74–80 (2009)Google Scholar
  4. 4.
    Valstar, M.F., Pantic, M.: Fully automatic recognition of the temporal phases of facial actions. IEEE Transactions on Systems, Man and Cybernetics 42, 28–43 (2012)CrossRefGoogle Scholar
  5. 5.
    Koelstra, S., Pantic, M., Patras, I.: A dynamic texture based approach to recognition of facial actions and their temporal models. IEEE PAMI 32, 1940–1954 (2010)CrossRefGoogle Scholar
  6. 6.
    Pantic, M., Patras, I.: Detecting facial actions and their temporal segments in nearly frontal-view face image sequences. In: Proc. of IEEE Int’l Conf. Systems, Man and Cybernetics, pp. 3358–3363 (2005)Google Scholar
  7. 7.
    Pantic, M., Patras, I.: Dynamics of facial expression: Recognition of facial actions and their temporal segments from face profile image sequences. IEEE Transactions on Systems, Man and Cybernetics - Part B 36(2), 433–449 (2006)CrossRefGoogle Scholar
  8. 8.
    Kim, M., Pavlovic, V.: Structured Output Ordinal Regression for Dynamic Facial Emotion Intensity Prediction. In: Daniilidis, K. (ed.) ECCV 2010, Part III. LNCS, vol. 6313, pp. 649–662. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  9. 9.
    Rudovic, O., Pavlovic, V., Pantic, M.: Multi-output laplacian dynamic ordinal regression for facial expression recognition and intensity estimation. In: CVPR (in press, 2012)Google Scholar
  10. 10.
    Lafferty, J., McCallum, A., Pereira, F.: Conditional Random Fields: Probabilistic models for segmenting and labeling sequence data. In: ICML, pp. 282–289 (2001)Google Scholar
  11. 11.
    He, X., Niyogi, P.: Locality Preserving Projections. In: NIPS (2004)Google Scholar
  12. 12.
    Lafferty, J., Zhu, X., Liu, Y.: Kernel conditional random fields: representation and clique selection. In: ICML 2004, p. 64 (2004)Google Scholar
  13. 13.
    Barla, A., Odone, F.,, V.: Histogram intersection kernel for image classification. In: ICIP 2003, vol. 2, pp. III-513–III-516 (2003)Google Scholar
  14. 14.
    Jiang, B., Valstar, M.F., Pantic, M.: Action unit detection using sparse appearance descriptors in space-time video volumes. In: FG 2011, pp. 314–321 (2011)Google Scholar
  15. 15.
    Pantic, M., Valstar, M.F., Rademaker, R., Maat, L.: Web-based database for facial expression analysis. In: ICME 2005, pp. 317–321 (July 2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Ognjen Rudovic
    • 1
  • Vladimir Pavlovic
    • 2
  • Maja Pantic
    • 1
    • 3
  1. 1.Computing Dept.Imperial College LondonUK
  2. 2.Dept. of Computer ScienceRutgers UniversityUSA
  3. 3.EEMCSUniversity of TwenteThe Netherlands

Personalised recommendations