Skip to main content
SpringerLink
Log in

A survey on facial expression recognition in 3D video sequences

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

An Erratum to this article was published on 17 May 2014

Abstract

Facial expression recognition constitutes an active research area due to its various applications. This survey addresses methodologies for 3D mesh video facial expression recognition. Recognition is, actually, a special case of intra-class retrieval. The approaches are analyzed and compared in detail. They are primarily categorized according to the 3D dynamic face analysis technique used. In addition, currently available datasets, used for 3D video facial expression analysis, are presented. Finally, future challenges that can be addressed in order for 3D video facial expression recognition field to be further improved, are extensively discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29

Similar content being viewed by others

References

  1. Berretti S, Amor BB, Daoudi M, Bimbo AD (2010) Person independent 3D facial expression recognition by a selected ensemble of SIFT descriptors. In: 3DOR ’10. Eurographics Association, pp 47–54

  2. Berretti S, Amor BB, Daoudi M, Bimbo AD (2011) 3D facial expression recognition using SIFT descriptors of automatically detected keypoints. Vis Comput 27(11):1021–1036

    Article  Google Scholar 

  3. Berretti S, Bimbo AD, Pala P, Amor BB, Daoudi M (2010) A set of selected SIFT features for 3D facial expression recognition. In: ICPR ’10. IEEE, pp 4125–4128

  4. Berretti S, Del Bimbo A, Pala P (2012) Real-time expression recognition from dynamic sequences of 3D facial scans. In: EU Workshop on 3D object retrieval. pp 85–92

  5. Berretti S, Del Bimbo A, Pala P (2012) Superfaces: a super-resolution model for 3D faces. In: Computer Vision ECCV 2012. Workshops and Demonstrations, vol 7583. Springer, Berlin Heidelberg, pp 73–82

  6. Bourgain J (1985) On Lipschitz embedding of finite metric spaces in Hilbert space. Israel J Math 52(1):46–52

    Article  MATH  MathSciNet  Google Scholar 

  7. Canavan S, Yin L (2012) 3D feature tracking using a deformable shape model. In: Technical report. Binghamton University

  8. Canavan SJ, Sun Y, Zhang X, Yin L (2012) A dynamic curvature based approach for facial activity analysis in 3D space. In: CVPR workshops. pp 14–19

  9. Chang Y, Vieira MB, Turk M, Velho L (2005) Automatic 3D facial expression analysis in videos. In: IEEE workshop AMFG ’05. pp 293–307

  10. Cootes TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 23(6):681–685

    Article  Google Scholar 

  11. Cosker D, Krumhuber E, Hilton A (2011) A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling. In: Proceedings on ICCV ’11, pp 2296–2303

  12. Cristinacce D, Cootes T (2008) Automatic feature localisation with constrained local models. Pattern Recogn 41(10):3054–3067

    Article  MATH  Google Scholar 

  13. Danelakis A, Theoharis T, Pratikakis I (2012) 3D mesh video retrieval: a survey. In: Proceedings on 3DTV conference ’12. pp 1–4

  14. Drira H, Ben Amor B, Daoudi M, Srivastava A, Berretti S (2012) 3D dynamic expression recognition based on a novel deformation vector field and random forest. In: ICPR ’12. pp 1104–1107

  15. Ekman P, Friesen W (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto

    Google Scholar 

  16. Fang T, Zhao X, Ocegueda O, Shah SK, Kakadiaris IA (2011) 3D facial expression recognition: a perspective on promises and challenges. In: IEEE Proceedings on FG ’11. pp 603–610

  17. Fang T, Zhao X, Ocegueda O, Shah SK, Kakadiaris IA (2012) 3D/4D facial expression analysis: an advanced annotated face model approach. Image Vis Comput 30(10):738–749

    Article  Google Scholar 

  18. Fang T, Zhao X, Shah SK, Kakadiaris IA (2011) 4D facial expression recognition. In: ICCV ’11. pp 1594–1601

  19. Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395

    Article  MathSciNet  Google Scholar 

  20. Gkalelis N, Kim H, Hilton A, Nikolaidis N, Pitas I (2009) The i3DPost multi-view and 3D human action/interaction database. In: Proceedings on CVMP ’07. pp 159–168

  21. Johnson A (1997) Spin-Images: a representation for 3D surface matching. Ph.D. thesis, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA

  22. Kakadiaris IA, Passalis G, Toderici G, Murtuza MN, Lu Y, Karampatziakis N, Theoharis T (2007) Three-dimensional face recognition in the presence of facial expressions: an annotated deformable model approach. IEEE Trans Pattern Anal Mach Intell 29(4):640–649

    Article  Google Scholar 

  23. Kharevych L, Springborn B, Schröder P (2006) Discrete conformal mappings via circle patterns. ACM Trans Graph 25(2):412–438

    Article  Google Scholar 

  24. Kimmel R, Sethian JA (1998) Computing geodesic paths on manifolds. In: Proceedings on the national academy science USA. pp 8431–8435

  25. Lanitis A, Taylor C, Cootes T (1997) Automatic interpretation and coding of face images using flexible models. IEEE Trans Pattern Anal Mach Intell 19(7):743–756

    Article  Google Scholar 

  26. László AJ, András L, Tamás N, Zsolt P, Judit S, Zoltán S, Dániel T (2012) 3D shape estimation in video sequences provides high precision evaluation of facial expressions. Image Vis Comput 30(10):785–795

    Article  Google Scholar 

  27. Le V, Tang H, Huang TS (2011) Expression recognition from 3D dynamic faces using robust spatio-temporal shape features. In: IEEE FG ’11. pp 414–421

  28. Matuszewski B, Quan W, Shark L, McLoughlin A, Lightbody C, Emsley H, Watkins C (2012) Hi4D-ADSIP 3D dynamic facial articulation database. Elsevier Image Vis Comput 30(10):713–727

    Article  Google Scholar 

  29. Rosato M, Chen X, Yin L (2008) Automatic registration of vertex correspondences for 3D facial expression analysis. In: IEEE international conference on biometrics: theory, applications and systems. pp 1–7

  30. Rueckert D, Sonoda LI, Hayes C, Hill DLG, Leach MO, Hawkes DJ (1999) Nonrigid registration using free-form deformations: application to breast MR images. IEEE Trans Med Imaging 18(8):712–721

    Article  Google Scholar 

  31. Sandbach G, Zafeiriou S, Pantic M, Rueckert D (2011) A dynamic approach to the recognition of 3D facial expressions and their temporal models. In: IEEE FG ’11. pp 406–413

  32. Sandbach G, Zafeiriou S, Pantic M, Rueckert D (2012) Recognition of 3D facial expression dynamics. Elsevier Image Vis Comput 30(10):762–773

    Article  Google Scholar 

  33. Sandbach G, Zafeiriou S, Pantic M, Yin L (2012) Static and dynamic 3D facial expression recognition: a comprehensive survey. Image Vis Comput 30(10):683–697

    Article  Google Scholar 

  34. Saragih JM, Lucey S, Cohn JF (2011) Deformable model fitting by regularized landmark mean-shift. Int J Comput Vis 91(2):200–215

    Article  MATH  MathSciNet  Google Scholar 

  35. Starck J, Hilton A (2007) Surface capture for performance-based animation. IEEE Comput Graph Appl 27(3):21–31

    Article  Google Scholar 

  36. Sun Y, Chen X, Rosato MJ, Yin L (2010) Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis. IEEE Trans Syst Man Cybern Part A 40(3):461–474

    Article  Google Scholar 

  37. Sun Y, Reale M, Yin L (2008) Recognizing partial facial action units based on 3D dynamic range data for facial expression recognition. In: FG ’08. pp 1–8

  38. Sun Y, Yin L (2008) Facial expression recognition based on 3D dynamic range model sequences. In: Springer proceedings on ECCV ’08: part II, pp 58–71

  39. Tsalakanidou F, Malassiotis S (2009) Robust facial action recognition from real-time 3D streams. In: CVPR ’09. pp 4–11

  40. Tsalakanidou F, Malassiotis S (2010) Real-time 2D + 3D facial action and expression recognition. Elsevier Pattern Recogn 43(5):1763–1775

    Article  Google Scholar 

  41. Yin L, Basu A (2001) Generating realistic facial expressions with wrinkles for model-based coding. Comput Vis Image Underst 84(2):201–240

    Article  MATH  Google Scholar 

  42. Yin L, Chen X, Sun Y, Worm T, Reale M (2008) A high-resolution 3D dynamic facial expression database. In: IEEE Proceedings on FG ’08. pp 1–6

  43. Yin L, Wei X, Longo P, Bhuvanesh A (2006) Analyzing facial expressions using intensity-variant 3D data for human computer interaction. In: Proceedings on ICPR ’06. pp 1248–1251

  44. Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3D facial expression database for facial behavior research. In: IEEE Proceedings on FGR ’06. pp 211–216

  45. Zaharescu A, Boyer E, Varanasi K, Horaud R (2009) Surface feature detection and description with applications to mesh matching. In: Proceedings on IEEE computer society conference on computer vision and pattern recognition. pp 373–380

  46. Zhang X, Reale M, Yin L (2013) Nebula feature: a space-time feature for posed and spontaneous 4D facial behavior analysis. In: IEEE FG ’13

  47. Zhang X, Yin L, Cohn JF, Canavan S, Reale M, Horowitz A, Liu P (2013) A high-resolution spontaneous 3D dynamic facial expression database. In: IEEE FG ’13

  48. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928

    Article  Google Scholar 

Download references

Acknowledgments

This research has been co-financed by the European Union (European Social Fund - ESF) and Greek national funds through the Operational Program “Education and Lifelong Learning” of the National Strategic Reference Framework (NSRF) - Research Funding Program: THALES-3DOR (MIS 379516).

Author information

Authors and Affiliations

  1. Department of Informatics and Telecommunications, National and Kapodistrian University of Athens, Athens, Greece

    Antonios Danelakis & Theoharis Theoharis

  2. Department of Computer and Information Science, Norwegian University of Science and Technology, Trondheim, Norway

    Theoharis Theoharis

  3. Department of Electrical and Computer Engineering, Democritus University of Thrace, Thrace, Greece

    Ioannis Pratikakis

Authors
  1. Antonios Danelakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Theoharis Theoharis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ioannis Pratikakis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Antonios Danelakis.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Danelakis, A., Theoharis, T. & Pratikakis, I. A survey on facial expression recognition in 3D video sequences. Multimed Tools Appl 74, 5577–5615 (2015). https://doi.org/10.1007/s11042-014-1869-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-014-1869-6

Keywords

Search

Navigation