Robust Incremental Hidden Conditional Random Fields for Human Action Recognition

  • Michalis VrigkasEmail author
  • Ermioni Mastora
  • Christophoros Nikou
  • Ioannis A. Kakadiaris
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11241)


Hidden conditional random fields (HCRFs) are a powerful supervised classification system, which is able to capture the intrinsic motion patterns of a human action. However, finding the optimal number of hidden states remains a severe limitation for this model. This paper addresses this limitation by proposing a new model, called robust incremental hidden conditional random field (RI-HCRF). A hidden Markov model (HMM) is created for each observation paired with an action label and its parameters are defined by the potentials of the original HCRF graph. Starting from an initial number of hidden states and increasing their number incrementally, the Viterbi path is computed for each HMM. The method seeks for a sequence of hidden states, where each variable participates in a maximum number of optimal paths. Thereby, variables with low participation in optimal paths are rejected. In addition, a robust mixture of Student’s t-distributions is imposed as a regularizer to the parameters of the model. The experimental results on human action recognition show that RI-HCRF successfully estimates the number of hidden states and outperforms all state-of-the-art models.


Student’s t-distribution Hidden conditional random fields Hidden Markov model Action recognition 



This work has been co-funded by the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH-CREATE-INNOVATE (project code: T1EDK04517) and by the UH Hugh Roy and Lillie Cranz Cullen Endowment Fund. The authors gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research. All statements of fact, opinion or conclusions contained herein are those of the authors and should not be construed as representing the official views or policies of the sponsors.


  1. 1.
    Bouchard, G.: Bias-variance tradeoff in hybrid generative-discriminative models. In: ICMLA, pp. 124–129 (2007)Google Scholar
  2. 2.
    Bousmalis, K., Zafeiriou, S., Morency, L.P., Pantic, M.: Infinite hidden conditional random fields for human behavior analysis. Trans. Neural Netw. Learn. Syst. 24(1), 170–177 (2013)CrossRefGoogle Scholar
  3. 3.
    Bousmalis, K., Zafeiriou, S., Morency, L.P., Pantic, M., Ghahramani, Z.: Variational hidden conditional random fields with coupled dirichlet process mixtures. In: Proceedings of Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 531–547 (2013)CrossRefGoogle Scholar
  4. 4.
    Byrd, R.H., Nocedal, J., Schnabel, R.B.: Representations of quasi-Newton matrices and their use in limited memory methods. Math. Program. 63(1), 129–156 (1994)MathSciNetCrossRefGoogle Scholar
  5. 5.
    Carreira, J., Zisserman, A.: Quo vadis, action recognition? A new model and the kinetics dataset. In: CVPR, pp. 6299–6308, July 2017Google Scholar
  6. 6.
    Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2, 1–27 (2011)CrossRefGoogle Scholar
  7. 7.
    Donahue, J., Hendricks, L.A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., Darrell, T.: Long-term recurrent convolutional networks for visual recognition and description. In: CVPR, pp. 2625–2634, June 2015Google Scholar
  8. 8.
    Feichtenhofer, C., Pinz, A., Zisserman, A.: Convolutional two-stream network fusion for video action recognition. In: CVPR, June 2016Google Scholar
  9. 9.
    Lafferty, J.D., Pereira, F.C.N., McCallum, A.: Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: ICML, pp. 282–289 (2001)Google Scholar
  10. 10.
    Laptev, I.: On space-time interest points. Int. J. Comput. Vis. 64(2–3), 107–123 (2005)CrossRefGoogle Scholar
  11. 11.
    McLachlan, G.J., Peel, D.: Robust mixture modelling using the t distribution. Stat. Comput. 10(4), 335–344 (2000)Google Scholar
  12. 12.
    Patron-Perez, A., Marszalek, M., Reid, I., Zisserman, A.: Structured learning of human interactions in TV shows. IEEE Trans. Pattern Anal. Mach. Intell. 34(12), 2441–2453 (2012)CrossRefGoogle Scholar
  13. 13.
    Quattoni, A., Wang, S., Morency, L.P., Collins, M., Darrell, T.: Hidden conditional random fields. IEEE Trans. Pattern Anal. Mach. Intell. 29(10), 1848–1852 (2007)CrossRefGoogle Scholar
  14. 14.
    Rabiner, L., Juang, B.H.: Fundamentals of Speech Recognition. Prentice-Hall, Englewood Cliffs (1993)Google Scholar
  15. 15.
    Sigurdsson, G.A., Russakovsky, O., Gupta, A.: What actions are needed for understanding human actions in videos? In: Proceedings of IEEE International Conference on Computer Vision, pp. 2156–2165 (2017)Google Scholar
  16. 16.
    Song, Y., Morency, L.P., Davis, R.: Multi-view latent variable discriminative models for action recognition. In: CVPR, Providence, RI, June 2012Google Scholar
  17. 17.
    Soullard, Y., Artières, T.: Hybrid HMM and HCRF model for sequence classification. In: ESANN, pp. 453–458 (2011)Google Scholar
  18. 18.
    Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3D convolutional networks. In: ICCV, pp. 4489–4497 (2015)Google Scholar
  19. 19.
    Vrigkas, M., Kazakos, E., Nikou, C., Kakadiaris, I.A.: Inferring human activities using robust privileged probabilistic learning. In: ICCVW, pp. 2658–2665 (2017)Google Scholar
  20. 20.
    Vrigkas, M., Nikou, C., Kakadiadis, I.A.: Classifying behavioral attributes using conditional random fields. In: Likas, A., Blekas, K., Kalles, D. (eds.) SETN 2014. LNCS (LNAI), vol. 8445, pp. 95–104. Springer, Cham (2014). Scholar
  21. 21.
    Vrigkas, M., Nikou, C., Kakadiaris, I.A.: A review of human activity recognition methods. Front. Robot. AI 2(28), 1–26 (2015)Google Scholar
  22. 22.
    Vrigkas, M., Nikou, C., Kakadiaris, I.A.: Identifying human behaviors using synchronized audio-visual cues. IEEE Trans. Affect. Comput. 8(1), 54–66 (2017)CrossRefGoogle Scholar
  23. 23.
    Wang, L., Xiong, Y., Lin, D., Van Gool, L.: Untrimmednets for weakly supervised action recognition and detection. In: CVPR, pp. 4325–4334, July 2017Google Scholar
  24. 24.
    Yun, K., Honorio, J., Chattopadhyay, D., Berg, T.L., Samaras, D.: Two-person interaction detection using body-pose features and multiple instance learning. In: CVPRW, pp. 28–35 (2012)Google Scholar
  25. 25.
    Zhang, J., Gong, S.: Action categorization with modified hidden conditional random field. Pattern Recognit. 43(1), 197–203 (2010)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Michalis Vrigkas
    • 1
    Email author
  • Ermioni Mastora
    • 2
  • Christophoros Nikou
    • 2
  • Ioannis A. Kakadiaris
    • 1
  1. 1.Computational Biomedicine Laboratory, Department of Computer ScienceUniversity of HoustonHoustonUSA
  2. 2.Department of Computer Science and EngineeringUniversity of IoanninaIoanninaGreece

Personalised recommendations