Dynamic Probabilistic CCA for Analysis of Affective Behaviour

  • Mihalis A. Nicolaou
  • Vladimir Pavlovic
  • Maja Pantic
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7578)


Fusing multiple continuous expert annotations is a crucial problem in machine learning and computer vision, particularly when dealing with uncertain and subjective tasks related to affective behaviour. Inspired by the concept of inferring shared and individual latent spaces in probabilistic CCA (PCCA), we firstly propose a novel, generative model which discovers temporal dependencies on the shared/individual spaces (DPCCA). In order to accommodate for temporal lags which are prominent amongst continuous annotations, we further introduce a latent warping process. We show that the resulting model (DPCTW) (i) can be used as a unifying framework for solving the problems of temporal alignment and fusion of multiple annotations in time, and (ii) that by incorporating dynamics, modelling annotation/sequence specific biases, noise estimation and time warping, DPCTW outperforms state-of-the-art methods for both the aggregation of multiple, yet imperfect expert annotations as well as the alignment of affective behavior.


Ground Truth Latent Space Dynamic Time Warping Observation Sequence Relevance Vector Machine 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Gunes, H., et al.: Emotion representation, analysis and synthesis in continuous space: A survey. In: Proc. of IEEE FG 2011 EmoSPACE WS, Santa Barbara, CA, USA, pp. 827–834 (2011)Google Scholar
  2. 2.
    Cowie, R., McKeown, G.: Statistical analysis of data from initial labelled database and recommendations for an economical coding scheme (2010),
  3. 3.
    Wöllmer, M., et al.: Abandoning emotion classes. In: INTERSPEECH, pp. 597–600 (2008)Google Scholar
  4. 4.
    Nicolaou, M.A., et al.: Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space. IEEE Trans. on Affective Computing 2, 92–105 (2011)CrossRefGoogle Scholar
  5. 5.
    Raykar, V.C., et al.: Learning from crowds. Journal of Machine Learning Research 11, 1297–1322 (2010)MathSciNetGoogle Scholar
  6. 6.
    Zhou, F., De la Torre, F.: Canonical time warping for alignment of human behavior. In: Advances in Neural Information Processing Systems, vol. 22, pp. 2286–2294 (2009)Google Scholar
  7. 7.
    Klami, A., Kaski, S.: Probabilistic approach to detecting dependencies between data sets. Neurocomput. 72, 39–46 (2008)CrossRefGoogle Scholar
  8. 8.
    Bach, F.R., Jordan, M.I.: A Probabilistic Interpretation of Canonical Correlation Analysis. Technical report, University of California, Berkeley (2005)Google Scholar
  9. 9.
    Zeng, Z., et al.: A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Trans. PAMI. 31, 39–58 (2009)CrossRefGoogle Scholar
  10. 10.
    Kim, M., Pavlovic, V.: Discriminative Learning for Dynamic State Prediction. IEEE Trans. PAMI. 31, 1847–1861 (2009)CrossRefGoogle Scholar
  11. 11.
    Ghahramani, Z., Roweis, S.T.: Learning nonlinear dynamical systems using an EM algorithm. In: Advances in NIPS, pp. 599–605. MIT Press (1999)Google Scholar
  12. 12.
    Roweis, S., Ghahramani, Z.: A unifying review of linear Gaussian models. Neural Computation 11, 305–345 (1999)CrossRefGoogle Scholar
  13. 13.
    Ghahramani, Z., Jordan, M.I., Smyth, P.: Factorial hidden markov models. In: Machine Learning, vol. 29, pp. 245–273. MIT Press (1997)Google Scholar
  14. 14.
    Van der Merwe, R., Wan, E.: The square-root unscented Kalman filter for state and parameter-estimation. In: Proc. of IEEE ICASP 2001, vol. 6, pp. 3461–3464 (2001)Google Scholar
  15. 15.
    McKeown, G., et al.: The SEMAINE corpus of emotionally coloured character interactions. In: ICME, pp. 1079–1084 (2010)Google Scholar
  16. 16.
    Rabiner, L., Juang, B.H.: Fundamentals of Speech Recognition. United states edn. Prentice-Hall (1993)Google Scholar
  17. 17.
    Patras, I., Pantic, M.: Particle filtering with factorized likelihoods for tracking facial features. In: Proc. of IEEE FG 2004, pp. 97–102 (2004)Google Scholar
  18. 18.
    Tipping, M.E.: Sparse Bayesian Learning and the Relevance Vector Machine. Journal of Machine Learning Research 1, 211–244 (2001)MathSciNetzbMATHGoogle Scholar
  19. 19.
    Pantic, M., et al.: Web-based database for facial expression analysis. In: Proc. of IEEE ICME, Amsterdam, The Netherlands, pp. 317–321 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Mihalis A. Nicolaou
    • 1
  • Vladimir Pavlovic
    • 2
  • Maja Pantic
    • 1
    • 3
  1. 1.Dept. of ComputingImperial College LondonUK
  2. 2.Dept. of Computer ScienceRutgers UniversityUSA
  3. 3.EEMCSUniversity of TwenteThe Netherlands

Personalised recommendations