Skip to main content
SpringerLink
Log in

Action unit detection in 3D facial videos with application in facial expression retrieval and recognition

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This work introduces a new scheme for action unit detection in 3D facial videos. Sets of features that define action unit activation in a robust manner are proposed. These features are computed based on eight detected facial landmarks on each facial mesh that involve angles, areas and distances. Support vector machine classifiers are then trained using the above features in order to perform action unit detection. The proposed AU detection scheme is used in a dynamic 3D facial expression retrieval and recognition pipeline, highlighting the most important AU s, in terms of providing facial expression information, and at the same time, resulting in better performance than state-of-the-art methodologies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Ashraf AB, Lucey S, Cohn JF, Chen T, Ambadar Z, Prkachin KM, Solomon PE (2009) The painful face – pain expression recognition using active appearance models. Image Vis Comput 27(12):1788–1796

    Article  Google Scholar 

  2. Baltrušaitis T, Mahmoud M, Robinson P (2015) Cross-dataset learning and person-specific normalisation for automatic action unit detection. In: Automatic face and gesture recognition (FG), pp 1–6

  3. Bartlett M, Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2006) Fully automatic facial action recognition in spontaneous behavior. In: 7th international conference on automatic face and gesture recognition (FGR), pp 223–230

  4. Bayramoglu N, Zhao G, Pietikäinen M (2013) CS-3DLBP And geometry based person independent 3d facial action unit detection. In: International conference on biometrics, pp 1–6

  5. Berretti S, Del Bimbo A, Pala P (2013) Automatic facial expression recognition in real-time from dynamic sequences of 3D face scans. Vis Comput 29(12):1333–1350

    Article  Google Scholar 

  6. Canavan SJ, Sun Y, Zhang X, Yin L (2012) A dynamic curvature based approach for facial activity analysis in 3D space. In: CVPR workshops, pp 14–19

  7. Chang Y, Vieira MB, Turk M, Velho L (2005) Automatic 3D facial expression analysis in videos. In: IEEE workshop AMFG ’05, pp 293–307

  8. Chen J, Liu X, Tu P, Aragones A (2013) Learning person-specific models for facial expression and action unit recognition. Pattern Recogn Lett 34(15):1964–1970

    Article  Google Scholar 

  9. Choi SS, Cha SH (2010) A survey of binary similarity and distance measures. Journal of Systemics Cybernetics and Informatics 8(1):43–48

    Google Scholar 

  10. Chu WS, De La Torre F, Cohn JF (2013) Selective transfer machine for personalized facial action unit detection. In: The IEEE conference on computer vision and pattern recognition (CVPR), pp 3515–3522

  11. Cosker D, Krumhuber E, Hilton A (2011) A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling. In: Proc. ICCV ’11, pp 2296–2303

  12. Dahmane M, Meunier J (2014) Prototype-based modeling for facial expression analysis. IEEE Trans Multimedia 16(6):1574–1584

    Article  Google Scholar 

  13. Danelakis A, Theoharis T, Pratikakis I (2014) Geotopo: dynamic 3D facial expression retrieval using topological and geometric information. In: Proc. 3d object retrieval 2014, pp 1–8

  14. Danelakis A, Theoharis T, Pratikakis I, Perakis P (2016) An effective methodology for dynamic 3D facial expression retrieval. Pattern Recogn 52:174–185

    Article  Google Scholar 

  15. Dapogny A, Bailly K, Dubuisson S (2016) Confidence-weighted local expression predictions for occlusion handling in expression recognition and action unit detection. Int J Comput Vis 126(2-4):255–271

    Article  MathSciNet  Google Scholar 

  16. Devijver PA, Kittler J (1982) Pattern recognition: a statistical approach. Prentice Hall, Englewood Cliffs

    MATH  Google Scholar 

  17. Donato G, Bartlett MS, Hager JC, Ekman P, Sejnowski TJ (1999) Classifying facial actions. IEEE Trans Pattern Anal Mach Intell 21(10):974–989

    Article  Google Scholar 

  18. Drira H, Ben Amor B, Daoudi M, Berretti S (2013) A dense deformation field for facial expression analysis in dynamic sequences of 3D scans. Springer International Publishing, Berlin, pp 148–159

    Google Scholar 

  19. Ekman P, Friesen W (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto

    Google Scholar 

  20. Fang T, Zhao X, Shah SK, Kakadiaris IA (2011) 4D facial expression recognition. In: ICCV ’11, pp 1594–1601

  21. Fang T, Zhao X, Ocegueda O, Shah SK, Kakadiaris IA (2012) 3d/4D facial expression analysis: an advanced annotated face model approach. Image Vis Comput 30(10):738–749

    Article  Google Scholar 

  22. Fasel B, Luettin J (1999) Automatic facial expression analysis: a survey. Pattern Recogn 36(1):259–275

    Article  MATH  Google Scholar 

  23. Gao G, Xu M, Shen J, Ma H, Yan S (2016) Cast2face: assigning character names onto faces in movie with actor-character correspondence. IEEE Trans Circuits Syst Video Technol 26(12):2299–2312

    Article  Google Scholar 

  24. Gehrig T, Ekenel H (2011) A common framework for real-time emotion recognition and facial action unit detection. In: IEEE computer society conference on computer vision and pattern recognition workshops (CVPRW), pp 1–6

  25. Gudi A, Tasli HE, den Uyl TM, Maroulis A (2015) Deep learning based facs action unit occurrence and intensity estimation. In: Automatic face and gesture recognition (FG), pp 1–5

  26. Huang Y, Li Y, Fan N (2010) Robust symbolic dual-view facial expression recognition with skin wrinkles: Local versus global approach. IEEE Trans Multimedia 12(6):536–543

    Article  Google Scholar 

  27. Jaiswal S, Valstar M (2016) Deep learning the dynamic appearance and shape of facial action units. In: 2016 IEEE winter conference on applications of computer vision (WACV), pp 1–8

  28. Jeni LA, Lórincz A, Nagy T, Palotai Z, Sebók J, Szabó Z, Takács D (2012) 3D shape estimation in video sequences provides high precision evaluation of facial expressions. Image Vis Comput 30(10):785–795

    Article  Google Scholar 

  29. Jeni LA, Girard JM, Cohn JF, Torre FDL (2013) Continuous au intensity estimation using localized, sparse facial feature space. In: 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), pp 1–7

  30. Jiang B, Valstar MF, Pantic M (2011) Action unit detection using sparse appearance descriptors in space-time video volumes. In: Automatic face gesture recognition and workshops, pp 314–321

  31. Khademi M, Morency LP (2014) Relative facial action unit detection. In: IEEE winter conference on applications of computer vision (WACV), pp 1090–1095

  32. Kotsia I, Zafeiriou S, Pitas I (2008) Texture and shape information fusion for facial expression and facial action unit recognition. Pattern Recogn 41(3):833–851

    Article  MATH  Google Scholar 

  33. Le V, Tang H, Huang TS (2011) Expression recognition from 3D dynamic faces using robust spatio-temporal shape features. In: IEEE FG ’11, pp 414–421

  34. Lien JJJ, Kanade T, Cohn J, Li CC (1998) Subtly different facial expression recognition and expression intensity estimation. In: IEEE conference on computer vison and pattern recogntion, pp 853–859

  35. Liu L, Cheng L, Liu Y, Jia Y, Rosenblum DS (2016) Recognizing complex activities by a probabilistic interval-based model. In: Proceedings of the thirtieth AAAI conference on artificial intelligence, pp 1266–1272

  36. Liu Y, Nie L, Han L, Zhang L, Rosenblum DS (2016) Action2activity: recognizing complex activities from sensor data. Computer Vision and Pattern Recognition. arXiv:1611.01872

  37. Liu Y, Nie L, Liu L, Rosenblum DS (2016) From action to activity: sensor-based activity recognition. Neurocomputing 181(Supplement C):108–115

    Article  Google Scholar 

  38. Lucey P, Cohn JF, Matthews I, Lucey S, Sridharan S, Howlett J, Prkachin KM (2011) Automatically detecting pain in video through facial action units. IEEE Trans Syst Man Cybern B Cybern 41(3):664–674

    Article  Google Scholar 

  39. Maalej A, Amor BB, Daoudi M, Srivastava A, Berretti S (2010) Local 3d shape analysis for facial expression recognition. In: 2010 20th international conference on pattern recognition, pp 4129–4132

  40. Maalej A, Amor BB, Daoudi M, Srivastava A, Berretti S (2011) Shape analysis of local facial patches for 3d facial expression recognition. Pattern Recogn 44 (8):1581–1589

    Article  Google Scholar 

  41. Mahoor MH, Zhou M, Veon KL, Mavadati SM, Cohn JF (2011) Facial action unit recognition with sparse representation. In: Automatic face gesture recognition and workshops, pp 336–342

  42. Mao Q, Rao Q, Yu Y, Dong M (2017) Hierarchical bayesian theme models for multipose facial expression recognition. IEEE Trans Multimedia 19(4):861–873

    Article  Google Scholar 

  43. Pantic M, Patras I (2006) Dynamics of facial expression: recognition of facial actions and their temporal segments from face profile image sequences. IEEE Trans Syst Man Cybern B Cybern 36(2):433–449

    Article  Google Scholar 

  44. Pantic M, Rothkrantz LJM (2000) Automatic analysis of facial expressions: the state of the art. IEEE Trans Pattern Anal Mach Intell 22(12):1424–1445

    Article  Google Scholar 

  45. Pantic M, Rothkrantz LJM (2004) Facial action recognition for facial expression analysis from static face images. IEEE Trans Syst Man Cybern B Cybern 34(3):1449–1461

    Article  Google Scholar 

  46. Perakis P, Theoharis T, Kakadiaris IA (2014) Feature fusion for facial landmark detection. Pattern Recogn 47(9):2783–2793

    Article  Google Scholar 

  47. Pinto SCD, Mena-Chalco JP, Lopes FM, Velho L (2011) Cesar, r.: 3d facial expression analysis by using 2d and 3d wavelet transforms. In: 2011 18th IEEE international conference on image processing (ICIP). IEEE, pp 1281–1284

  48. Ramanathan S, Kassim A, Venkatesh YV, Wah WS (2006) Human facial expression recognition using a 3d morphable model. In: 2006 International conference on image processing, pp 661–664

  49. Reale M, Zhang X, Yin L (2013) Nebula feature: a space-time feature for posed and spontaneous 4D facial behavior analysis. In: 10th IEEE international conference and workshops on automatic face and gesture recognition (FG), pp 1–8

  50. Rosato M, Chen X, Yin L (2008) Automatic registration of vertex correspondences for 3D facial expression analysis. In: IEEE international conference on biometrics: theory, applications and systems, pp 1–7

  51. Ruiz A, Van de Weijer J, Binefa X (2015) From emotions to action units with hidden and semi-hidden-task learning. In: The IEEE international conference on computer vision (ICCV), pp 3703–3711

  52. Sandbach G, Zafeiriou S, Pantic M (2012) Binary pattern analysis for 3D facial action unit detection. In: Proceedings of the british machine vision conference, BMVC 2012, Surrey, UK, pp 119.1–119.12

  53. Sandbach G, Zafeiriou S, Pantic M (2012) Local normal binary patterns for 3D facial action unit detection. In: 19th IEEE international conference on image processing (ICIP), pp 1813–1816

  54. Sandbach G, Zafeiriou S, Pantic M, Rueckert D (2012) Recognition of 3D facial expression dynamics. Elsevier Image and Vision Computing 30(10):762–773

    Article  Google Scholar 

  55. Savran A, Sankur B (2009) Automatic detection of facial actions from 3d data. In: 2009 IEEE 12th international conference on computer vision workshops, ICCV workshops, pp 1993–2000

  56. Savran A, Sankur B (2010) Detecting action units on 3d faces. In: 2010 IEEE 18th signal processing and communications applications conference, pp 300–303

  57. Savran A, Sankur B (2011) Detecting 3d facial action units via registration. In: 2011 IEEE 19th signal processing and communications applications conference (SIU), pp 371–374

  58. Savran A, Sankur B, Bilge MT (2012) Comparative evaluation of 3d vs. 2d modality for automatic detection of facial action units. Pattern Recogn 45(2):767–782

    Article  Google Scholar 

  59. Senechal T, Bailly K, Prevost L (2014) Impact of action unit detection in automatic emotion recognition. Pattern Anal Applic 17(1):51–67

    Article  MathSciNet  Google Scholar 

  60. Senechal T, McDuff D, el Kaliouby R (2015) Facial action unit detection using active learning and an efficient non-linear kernel approximation. In: The IEEE international conference on computer vision (ICCV) workshops, pp 10–18

  61. Senechal T, Rapp V, Salam H, Seguier R, Bailly K, Prevost L (2011) Combining aam coefficients with lgbp histograms in the multi-kernel svm framework to detect facial action units. In: 2011 IEEE international conference on automatic face & gesture recognition and workshops (FG 2011). IEEE, pp 860–865

  62. Senechal T, Rapp V, Salam H, Seguier R, Bailly K, Prevost L (2012) Facial action recognition combining heterogeneous features via multikernel learning. IEEE Trans Syst Man Cybern B Cybern 42(4):993–1005

    Article  Google Scholar 

  63. Simon T, Nguyen MH, Torre FDL, Cohn JF (2010) Action unit detection with segment-based svms. In: 2010 IEEE conference on computer vision and pattern recognition (CVPR), pp 2737–2744

  64. Song Y, McDuff D, Vasisht D, Kapoor A (2015) Exploiting sparsity and co-occurrence structure for action unit recognition. In: 2015 11th IEEE international conference and workshops on automatic face and gesture recognition (FG), pp 1–8

  65. Soyel H, Demirel H (2010) Optimal feature selection for 3d facial expression recognition using coarse-to-fine classification. Turk J Electr Eng Comput Sci 18 (6):1031–1040

    Google Scholar 

  66. Sun Y, Yin L (2008) Facial expression recognition based on 3D dynamic range model sequences. In: Springer proc. ECCV ’08: part II, pp 58–71

  67. Sun Y, Reale M, Yin L (2008) Recognizing partial facial action units based on 3d dynamic range data for facial expression recognition. In: Automatic face gesture recognition, pp 1–8

  68. Sun Y, Reale M, Yin L (2008) Recognizing partial facial action units based on 3D dynamic range data for facial expression recognition. In: FG ’08, pp 1–8

  69. Sun Y, Chen X, Rosato MJ, Yin L (2010) Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis. IEEE Trans Syst Man Cybern Syst Hum 40(3):461–474

    Article  Google Scholar 

  70. Tang H, Huang TS (2008) 3d facial expression recognition based on properties of line segments connecting facial feature points. In: 2008 8th IEEE international conference on automatic face gesture recognition, pp 1–6

  71. Tao D, Song M, Li X, Shen J, Sun J, Wu X, Faloutsos C, Maybank SJ (2008) Bayesian tensor approach for 3-d face modeling. IEEE Trans Circuits Syst Video Technol 18(10):1397–1410

    Article  Google Scholar 

  72. Tawari A, Trivedi MM (2013) Face expression recognition by cross modal data association. IEEE Trans Multimedia 15(7):1543–1552

    Article  Google Scholar 

  73. Tian YI, Kanade T, Cohn JF (2001) Recognizing action units for facial expression analysis. IEEE Trans Pattern Anal Mach Intell 23(2):97–115

    Article  Google Scholar 

  74. Tong Y, Liao W, Ji Q (2007) Facial action unit recognition by exploiting their dynamic and semantic relationships. IEEE Trans Pattern Anal Mach Intell 29 (10):1683–1699

    Article  Google Scholar 

  75. Tsalakanidou F, Malassiotis S (2009) Robust facial action recognition from real-time 3D streams. In: CVPR ’09, pp 4–11

  76. Tsalakanidou F, Malassiotis S (2010) Real-time 2d + 3D facial action and expression recognition. Pattern Recogn 43(5):1763–1775

    Article  Google Scholar 

  77. Tulyakov S, Vieriu RL, Sangineto E, Sebe N (2015) Facecept3d: real time 3d face tracking and analysis. In: The IEEE international conference on computer vision (ICCV) workshops, pp 29–33

  78. Valstar M, Pantic M (2006) Fully automatic facial action unit detection and temporal analysis. In: Computer vision and pattern recognition workshop (CVPRW), pp 149–149

  79. Valstar MF, Almaev T, Girard JM, McKeown G, Mehu M, Yin L, Pantic M, Cohn JF (2015) Fera 2015 - second facial expression recognition and analysis challenge. In: Automatic face and gesture recognition (FG), pp 1–8

  80. Valstar MF, Sánchez-Lozano E, Cohn JF, Jeni LA, Girard JM, Zhang Z, Yin L, Pantic M (2017) Fera 2017-addressing head pose in the third facial expression recognition and analysis challenge. arXiv:1702.04174

  81. van der Maaten L, Hendriks E (2012) Action unit classification using active appearance models and conditional random fields. Cogn Process 13(2):507–518

    Article  Google Scholar 

  82. Venkatesh YV, Kassim AK, Murthy OVR (2010) Resampling approach to facial expression recognition using 3d meshes. In: 2010 20th international conference on pattern recognition, pp 3772–3775

  83. Walecki R, Rudovic O, Pavlovic V, Pantic M (2015) Variable-state latent conditional random fields for facial expression recognition and action unit detection. In: Automatic face and gesture recognition, vol 1, pp 1–8

  84. Wegrzyn M, Vogt M, Kireclioglu B, Schneider J, Kissler J (2017) Mapping the emotional face. How individual face parts contribute to successful emotion recognition. PLOS ONE 12(5):1–15. https://doi.org/10.1371/journal.pone.0177239

    Article  Google Scholar 

  85. Wu CH, Wei WL, Lin JC, Lee WY (2013) Speaking effect removal on emotion recognition from facial expressions based on eigenface conversion. IEEE Trans Multimedia 15(8):1732–1744

    Article  Google Scholar 

  86. Xie L, Shen J, Han J, Zhu L, Shao L (2017) Dynamic multi-view hashing for online image retrieval. In: Proceedings of the twenty-sixth international joint conference on artificial intelligence, IJCAI-17, pp 3133–3139

  87. Yan J, Zheng W, Xu Q, Lu G, Li H, Wang B (2016) Sparse kernel reduced-rank regression for bimodal emotion recognition from facial expression and speech. IEEE Trans Multimedia 18(7):1319–1329

    Article  Google Scholar 

  88. Yüce A, Gao H, Thiran JP (2015) Discriminant multi-label manifold embedding for facial action unit detection. In: Automatic face and gesture recognition (FG), pp 1–6

  89. Yeasin M, Bullot B, Sharma R (2006) Recognition of facial expressions and measurement of levels of interest from video. IEEE Trans Multimedia 8(3):500–508

    Article  Google Scholar 

  90. Yin L, Wei X, Longo P, Bhuvanesh A (2006) Analyzing facial expressions using intensity-variant 3D data for human computer interaction. In: Proc. ICPR ’06, pp 1248–1251

  91. Yin L, Chen X, Sun Y, Worm T, Reale M (2008) A high-resolution 3D dynamic facial expression database. In: IEEE Proc. FG ’08, pp 1–6

  92. Yudin E, Wetzler A, Sela M, Kimmel R (2015) Improving 3d facial action unit detection with intrinsic normalization. In: Proceedings of the 1st international workshop on DIFFerential geometry in computer vision for analysis of shapes, images and trajectories (DIFF-CV 2015), 5, pp 1–10

  93. Zafeiriou S, Pitas I (2008) Discriminant graph structures for facial expression recognition. IEEE Trans Multimedia 10(8):1528–1540

    Article  Google Scholar 

  94. Zen G, Porzi L, Sangineto E, Ricci E, Sebe N (2016) Learning personalized models for facial expression analysis and gesture recognition. IEEE Trans Multimedia 18(4):775–788

    Article  Google Scholar 

  95. Zeng J, Chu WS, Torre FDl, Cohn JF, Xiong Z (2015) Confidence preserving machine for facial action unit detection. In: 2015 IEEE international conference on computer vision (ICCV), pp 3622–3630

  96. Zhang X, Reale M, Yin L (2013) Nebula feature: a space-time feature for posed and spontaneous 4D facial behavior analysis. In: IEEE FG ’13

  97. Zhang X, Yin L, Cohn JF, Canavan S, Reale M, Horowitz A, Liu P, Girard J (2014) BP4d spontaneous: a high-resolution spontaneous 3D dynamic facial expression database. Image Vis Comput 32(10):692–706

    Article  Google Scholar 

  98. Zhang T, Zheng W, Cui Z, Zong Y, Yan J, Yan K (2016) A deep neural network-driven feature learning method for multi-view facial expression recognition. IEEE Trans Multimedia 18(12):2528–2536

    Article  Google Scholar 

  99. Zhao X, Dellandréa E, Chen L, Samaras D (2010) Au recognition on 3d faces based on an extended statistical facial feature model. In: 2010 fourth IEEE international conference on biometrics: theory applications and systems (BTAS). IEEE, pp 1–6

  100. Zhao X, Dellandréa E, Zou J, Chen L (2013) A unified probabilistic framework for automatic 3d facial expression analysis based on a bayesian belief inference and statistical feature models. Image Vis Comput 31(3):231–245

    Article  Google Scholar 

  101. Zhao K, Chu WS, De la Torre F, Cohn JF, Zhang H (2015) Joint patch and multi-label learning for facial action unit detection. In: The IEEE conference on computer vision and pattern recognition (CVPR)

  102. Zhen Q, Huang D, Wang Y, Chen L (2016) Muscular movement model-based automatic 3d/4d facial expression recognition. IEEE Trans Multimedia 18(7):1438–1450

    Article  Google Scholar 

  103. Zhu Y, la Torre FD, Cohn JF, Zhang YJ (2009) Dynamic cascades with bidirectional bootstrapping for spontaneous facial action unit detection. In: 2009 3rd international conference on affective computing and intelligent interaction and workshops, pp 1–8

Download references

Author information

Authors and Affiliations

  1. Department of Computer & Information Science, Norwegian University of Science and Technology, 7034, Trondheim, Norway

    Antonios Danelakis & Theoharis Theoharis

  2. Department of Electrical & Computer Engineering, Democritus University of Thrace, 67100, Xanthi, Greece

    Ioannis Pratikakis

Authors
  1. Antonios Danelakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Theoharis Theoharis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ioannis Pratikakis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Antonios Danelakis.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Danelakis, A., Theoharis, T. & Pratikakis, I. Action unit detection in 3D facial videos with application in facial expression retrieval and recognition. Multimed Tools Appl 77, 24813–24841 (2018). https://doi.org/10.1007/s11042-018-5699-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-5699-9

Keywords

Search

Navigation