Multimedia Tools and Applications

, Volume 74, Issue 15, pp 5577–5615 | Cite as

A survey on facial expression recognition in 3D video sequences

  • Antonios Danelakis
  • Theoharis Theoharis
  • Ioannis Pratikakis


Facial expression recognition constitutes an active research area due to its various applications. This survey addresses methodologies for 3D mesh video facial expression recognition. Recognition is, actually, a special case of intra-class retrieval. The approaches are analyzed and compared in detail. They are primarily categorized according to the 3D dynamic face analysis technique used. In addition, currently available datasets, used for 3D video facial expression analysis, are presented. Finally, future challenges that can be addressed in order for 3D video facial expression recognition field to be further improved, are extensively discussed.


3D mesh video 3D dynamic facial meshes 3D video facial expression recognition 



This research has been co-financed by the European Union (European Social Fund - ESF) and Greek national funds through the Operational Program “Education and Lifelong Learning” of the National Strategic Reference Framework (NSRF) - Research Funding Program: THALES-3DOR (MIS 379516).


  1. 1.
    Berretti S, Amor BB, Daoudi M, Bimbo AD (2010) Person independent 3D facial expression recognition by a selected ensemble of SIFT descriptors. In: 3DOR ’10. Eurographics Association, pp 47–54Google Scholar
  2. 2.
    Berretti S, Amor BB, Daoudi M, Bimbo AD (2011) 3D facial expression recognition using SIFT descriptors of automatically detected keypoints. Vis Comput 27(11):1021–1036CrossRefGoogle Scholar
  3. 3.
    Berretti S, Bimbo AD, Pala P, Amor BB, Daoudi M (2010) A set of selected SIFT features for 3D facial expression recognition. In: ICPR ’10. IEEE, pp 4125–4128Google Scholar
  4. 4.
    Berretti S, Del Bimbo A, Pala P (2012) Real-time expression recognition from dynamic sequences of 3D facial scans. In: EU Workshop on 3D object retrieval. pp 85–92Google Scholar
  5. 5.
    Berretti S, Del Bimbo A, Pala P (2012) Superfaces: a super-resolution model for 3D faces. In: Computer Vision ECCV 2012. Workshops and Demonstrations, vol 7583. Springer, Berlin Heidelberg, pp 73–82Google Scholar
  6. 6.
    Bourgain J (1985) On Lipschitz embedding of finite metric spaces in Hilbert space. Israel J Math 52(1):46–52zbMATHMathSciNetCrossRefGoogle Scholar
  7. 7.
    Canavan S, Yin L (2012) 3D feature tracking using a deformable shape model. In: Technical report. Binghamton UniversityGoogle Scholar
  8. 8.
    Canavan SJ, Sun Y, Zhang X, Yin L (2012) A dynamic curvature based approach for facial activity analysis in 3D space. In: CVPR workshops. pp 14–19Google Scholar
  9. 9.
    Chang Y, Vieira MB, Turk M, Velho L (2005) Automatic 3D facial expression analysis in videos. In: IEEE workshop AMFG ’05. pp 293–307Google Scholar
  10. 10.
    Cootes TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 23(6):681–685CrossRefGoogle Scholar
  11. 11.
    Cosker D, Krumhuber E, Hilton A (2011) A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling. In: Proceedings on ICCV ’11, pp 2296–2303Google Scholar
  12. 12.
    Cristinacce D, Cootes T (2008) Automatic feature localisation with constrained local models. Pattern Recogn 41(10):3054–3067zbMATHCrossRefGoogle Scholar
  13. 13.
    Danelakis A, Theoharis T, Pratikakis I (2012) 3D mesh video retrieval: a survey. In: Proceedings on 3DTV conference ’12. pp 1–4Google Scholar
  14. 14.
    Drira H, Ben Amor B, Daoudi M, Srivastava A, Berretti S (2012) 3D dynamic expression recognition based on a novel deformation vector field and random forest. In: ICPR ’12. pp 1104–1107Google Scholar
  15. 15.
    Ekman P, Friesen W (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo AltoGoogle Scholar
  16. 16.
    Fang T, Zhao X, Ocegueda O, Shah SK, Kakadiaris IA (2011) 3D facial expression recognition: a perspective on promises and challenges. In: IEEE Proceedings on FG ’11. pp 603–610Google Scholar
  17. 17.
    Fang T, Zhao X, Ocegueda O, Shah SK, Kakadiaris IA (2012) 3D/4D facial expression analysis: an advanced annotated face model approach. Image Vis Comput 30(10):738–749CrossRefGoogle Scholar
  18. 18.
    Fang T, Zhao X, Shah SK, Kakadiaris IA (2011) 4D facial expression recognition. In: ICCV ’11. pp 1594–1601Google Scholar
  19. 19.
    Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395MathSciNetCrossRefGoogle Scholar
  20. 20.
    Gkalelis N, Kim H, Hilton A, Nikolaidis N, Pitas I (2009) The i3DPost multi-view and 3D human action/interaction database. In: Proceedings on CVMP ’07. pp 159–168Google Scholar
  21. 21.
    Johnson A (1997) Spin-Images: a representation for 3D surface matching. Ph.D. thesis, Robotics Institute, Carnegie Mellon University, Pittsburgh, PAGoogle Scholar
  22. 22.
    Kakadiaris IA, Passalis G, Toderici G, Murtuza MN, Lu Y, Karampatziakis N, Theoharis T (2007) Three-dimensional face recognition in the presence of facial expressions: an annotated deformable model approach. IEEE Trans Pattern Anal Mach Intell 29(4):640–649CrossRefGoogle Scholar
  23. 23.
    Kharevych L, Springborn B, Schröder P (2006) Discrete conformal mappings via circle patterns. ACM Trans Graph 25(2):412–438CrossRefGoogle Scholar
  24. 24.
    Kimmel R, Sethian JA (1998) Computing geodesic paths on manifolds. In: Proceedings on the national academy science USA. pp 8431–8435Google Scholar
  25. 25.
    Lanitis A, Taylor C, Cootes T (1997) Automatic interpretation and coding of face images using flexible models. IEEE Trans Pattern Anal Mach Intell 19(7):743–756CrossRefGoogle Scholar
  26. 26.
    László AJ, András L, Tamás N, Zsolt P, Judit S, Zoltán S, Dániel T (2012) 3D shape estimation in video sequences provides high precision evaluation of facial expressions. Image Vis Comput 30(10):785–795CrossRefGoogle Scholar
  27. 27.
    Le V, Tang H, Huang TS (2011) Expression recognition from 3D dynamic faces using robust spatio-temporal shape features. In: IEEE FG ’11. pp 414–421Google Scholar
  28. 28.
    Matuszewski B, Quan W, Shark L, McLoughlin A, Lightbody C, Emsley H, Watkins C (2012) Hi4D-ADSIP 3D dynamic facial articulation database. Elsevier Image Vis Comput 30(10):713–727CrossRefGoogle Scholar
  29. 29.
    Rosato M, Chen X, Yin L (2008) Automatic registration of vertex correspondences for 3D facial expression analysis. In: IEEE international conference on biometrics: theory, applications and systems. pp 1–7Google Scholar
  30. 30.
    Rueckert D, Sonoda LI, Hayes C, Hill DLG, Leach MO, Hawkes DJ (1999) Nonrigid registration using free-form deformations: application to breast MR images. IEEE Trans Med Imaging 18(8):712–721CrossRefGoogle Scholar
  31. 31.
    Sandbach G, Zafeiriou S, Pantic M, Rueckert D (2011) A dynamic approach to the recognition of 3D facial expressions and their temporal models. In: IEEE FG ’11. pp 406–413Google Scholar
  32. 32.
    Sandbach G, Zafeiriou S, Pantic M, Rueckert D (2012) Recognition of 3D facial expression dynamics. Elsevier Image Vis Comput 30(10):762–773CrossRefGoogle Scholar
  33. 33.
    Sandbach G, Zafeiriou S, Pantic M, Yin L (2012) Static and dynamic 3D facial expression recognition: a comprehensive survey. Image Vis Comput 30(10):683–697CrossRefGoogle Scholar
  34. 34.
    Saragih JM, Lucey S, Cohn JF (2011) Deformable model fitting by regularized landmark mean-shift. Int J Comput Vis 91(2):200–215zbMATHMathSciNetCrossRefGoogle Scholar
  35. 35.
    Starck J, Hilton A (2007) Surface capture for performance-based animation. IEEE Comput Graph Appl 27(3):21–31CrossRefGoogle Scholar
  36. 36.
    Sun Y, Chen X, Rosato MJ, Yin L (2010) Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis. IEEE Trans Syst Man Cybern Part A 40(3):461–474CrossRefGoogle Scholar
  37. 37.
    Sun Y, Reale M, Yin L (2008) Recognizing partial facial action units based on 3D dynamic range data for facial expression recognition. In: FG ’08. pp 1–8Google Scholar
  38. 38.
    Sun Y, Yin L (2008) Facial expression recognition based on 3D dynamic range model sequences. In: Springer proceedings on ECCV ’08: part II, pp 58–71Google Scholar
  39. 39.
    Tsalakanidou F, Malassiotis S (2009) Robust facial action recognition from real-time 3D streams. In: CVPR ’09. pp 4–11Google Scholar
  40. 40.
    Tsalakanidou F, Malassiotis S (2010) Real-time 2D + 3D facial action and expression recognition. Elsevier Pattern Recogn 43(5):1763–1775CrossRefGoogle Scholar
  41. 41.
    Yin L, Basu A (2001) Generating realistic facial expressions with wrinkles for model-based coding. Comput Vis Image Underst 84(2):201–240zbMATHCrossRefGoogle Scholar
  42. 42.
    Yin L, Chen X, Sun Y, Worm T, Reale M (2008) A high-resolution 3D dynamic facial expression database. In: IEEE Proceedings on FG ’08. pp 1–6Google Scholar
  43. 43.
    Yin L, Wei X, Longo P, Bhuvanesh A (2006) Analyzing facial expressions using intensity-variant 3D data for human computer interaction. In: Proceedings on ICPR ’06. pp 1248–1251Google Scholar
  44. 44.
    Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3D facial expression database for facial behavior research. In: IEEE Proceedings on FGR ’06. pp 211–216Google Scholar
  45. 45.
    Zaharescu A, Boyer E, Varanasi K, Horaud R (2009) Surface feature detection and description with applications to mesh matching. In: Proceedings on IEEE computer society conference on computer vision and pattern recognition. pp 373–380Google Scholar
  46. 46.
    Zhang X, Reale M, Yin L (2013) Nebula feature: a space-time feature for posed and spontaneous 4D facial behavior analysis. In: IEEE FG ’13Google Scholar
  47. 47.
    Zhang X, Yin L, Cohn JF, Canavan S, Reale M, Horowitz A, Liu P (2013) A high-resolution spontaneous 3D dynamic facial expression database. In: IEEE FG ’13Google Scholar
  48. 48.
    Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Antonios Danelakis
    • 1
  • Theoharis Theoharis
    • 1
    • 2
  • Ioannis Pratikakis
    • 3
  1. 1.Department of Informatics and TelecommunicationsNational and Kapodistrian University of AthensAthensGreece
  2. 2.Department of Computer and Information ScienceNorwegian University of Science and TechnologyTrondheimNorway
  3. 3.Department of Electrical and Computer EngineeringDemocritus University of ThraceThraceGreece

Personalised recommendations