Abstract
Facial expression recognition constitutes an active research area due to its various applications. This survey addresses methodologies for 3D mesh video facial expression recognition. Recognition is, actually, a special case of intra-class retrieval. The approaches are analyzed and compared in detail. They are primarily categorized according to the 3D dynamic face analysis technique used. In addition, currently available datasets, used for 3D video facial expression analysis, are presented. Finally, future challenges that can be addressed in order for 3D video facial expression recognition field to be further improved, are extensively discussed.
Similar content being viewed by others
References
Berretti S, Amor BB, Daoudi M, Bimbo AD (2010) Person independent 3D facial expression recognition by a selected ensemble of SIFT descriptors. In: 3DOR ’10. Eurographics Association, pp 47–54
Berretti S, Amor BB, Daoudi M, Bimbo AD (2011) 3D facial expression recognition using SIFT descriptors of automatically detected keypoints. Vis Comput 27(11):1021–1036
Berretti S, Bimbo AD, Pala P, Amor BB, Daoudi M (2010) A set of selected SIFT features for 3D facial expression recognition. In: ICPR ’10. IEEE, pp 4125–4128
Berretti S, Del Bimbo A, Pala P (2012) Real-time expression recognition from dynamic sequences of 3D facial scans. In: EU Workshop on 3D object retrieval. pp 85–92
Berretti S, Del Bimbo A, Pala P (2012) Superfaces: a super-resolution model for 3D faces. In: Computer Vision ECCV 2012. Workshops and Demonstrations, vol 7583. Springer, Berlin Heidelberg, pp 73–82
Bourgain J (1985) On Lipschitz embedding of finite metric spaces in Hilbert space. Israel J Math 52(1):46–52
Canavan S, Yin L (2012) 3D feature tracking using a deformable shape model. In: Technical report. Binghamton University
Canavan SJ, Sun Y, Zhang X, Yin L (2012) A dynamic curvature based approach for facial activity analysis in 3D space. In: CVPR workshops. pp 14–19
Chang Y, Vieira MB, Turk M, Velho L (2005) Automatic 3D facial expression analysis in videos. In: IEEE workshop AMFG ’05. pp 293–307
Cootes TF, Edwards GJ, Taylor CJ (2001) Active appearance models. IEEE Trans Pattern Anal Mach Intell 23(6):681–685
Cosker D, Krumhuber E, Hilton A (2011) A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling. In: Proceedings on ICCV ’11, pp 2296–2303
Cristinacce D, Cootes T (2008) Automatic feature localisation with constrained local models. Pattern Recogn 41(10):3054–3067
Danelakis A, Theoharis T, Pratikakis I (2012) 3D mesh video retrieval: a survey. In: Proceedings on 3DTV conference ’12. pp 1–4
Drira H, Ben Amor B, Daoudi M, Srivastava A, Berretti S (2012) 3D dynamic expression recognition based on a novel deformation vector field and random forest. In: ICPR ’12. pp 1104–1107
Ekman P, Friesen W (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto
Fang T, Zhao X, Ocegueda O, Shah SK, Kakadiaris IA (2011) 3D facial expression recognition: a perspective on promises and challenges. In: IEEE Proceedings on FG ’11. pp 603–610
Fang T, Zhao X, Ocegueda O, Shah SK, Kakadiaris IA (2012) 3D/4D facial expression analysis: an advanced annotated face model approach. Image Vis Comput 30(10):738–749
Fang T, Zhao X, Shah SK, Kakadiaris IA (2011) 4D facial expression recognition. In: ICCV ’11. pp 1594–1601
Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395
Gkalelis N, Kim H, Hilton A, Nikolaidis N, Pitas I (2009) The i3DPost multi-view and 3D human action/interaction database. In: Proceedings on CVMP ’07. pp 159–168
Johnson A (1997) Spin-Images: a representation for 3D surface matching. Ph.D. thesis, Robotics Institute, Carnegie Mellon University, Pittsburgh, PA
Kakadiaris IA, Passalis G, Toderici G, Murtuza MN, Lu Y, Karampatziakis N, Theoharis T (2007) Three-dimensional face recognition in the presence of facial expressions: an annotated deformable model approach. IEEE Trans Pattern Anal Mach Intell 29(4):640–649
Kharevych L, Springborn B, Schröder P (2006) Discrete conformal mappings via circle patterns. ACM Trans Graph 25(2):412–438
Kimmel R, Sethian JA (1998) Computing geodesic paths on manifolds. In: Proceedings on the national academy science USA. pp 8431–8435
Lanitis A, Taylor C, Cootes T (1997) Automatic interpretation and coding of face images using flexible models. IEEE Trans Pattern Anal Mach Intell 19(7):743–756
László AJ, András L, Tamás N, Zsolt P, Judit S, Zoltán S, Dániel T (2012) 3D shape estimation in video sequences provides high precision evaluation of facial expressions. Image Vis Comput 30(10):785–795
Le V, Tang H, Huang TS (2011) Expression recognition from 3D dynamic faces using robust spatio-temporal shape features. In: IEEE FG ’11. pp 414–421
Matuszewski B, Quan W, Shark L, McLoughlin A, Lightbody C, Emsley H, Watkins C (2012) Hi4D-ADSIP 3D dynamic facial articulation database. Elsevier Image Vis Comput 30(10):713–727
Rosato M, Chen X, Yin L (2008) Automatic registration of vertex correspondences for 3D facial expression analysis. In: IEEE international conference on biometrics: theory, applications and systems. pp 1–7
Rueckert D, Sonoda LI, Hayes C, Hill DLG, Leach MO, Hawkes DJ (1999) Nonrigid registration using free-form deformations: application to breast MR images. IEEE Trans Med Imaging 18(8):712–721
Sandbach G, Zafeiriou S, Pantic M, Rueckert D (2011) A dynamic approach to the recognition of 3D facial expressions and their temporal models. In: IEEE FG ’11. pp 406–413
Sandbach G, Zafeiriou S, Pantic M, Rueckert D (2012) Recognition of 3D facial expression dynamics. Elsevier Image Vis Comput 30(10):762–773
Sandbach G, Zafeiriou S, Pantic M, Yin L (2012) Static and dynamic 3D facial expression recognition: a comprehensive survey. Image Vis Comput 30(10):683–697
Saragih JM, Lucey S, Cohn JF (2011) Deformable model fitting by regularized landmark mean-shift. Int J Comput Vis 91(2):200–215
Starck J, Hilton A (2007) Surface capture for performance-based animation. IEEE Comput Graph Appl 27(3):21–31
Sun Y, Chen X, Rosato MJ, Yin L (2010) Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis. IEEE Trans Syst Man Cybern Part A 40(3):461–474
Sun Y, Reale M, Yin L (2008) Recognizing partial facial action units based on 3D dynamic range data for facial expression recognition. In: FG ’08. pp 1–8
Sun Y, Yin L (2008) Facial expression recognition based on 3D dynamic range model sequences. In: Springer proceedings on ECCV ’08: part II, pp 58–71
Tsalakanidou F, Malassiotis S (2009) Robust facial action recognition from real-time 3D streams. In: CVPR ’09. pp 4–11
Tsalakanidou F, Malassiotis S (2010) Real-time 2D + 3D facial action and expression recognition. Elsevier Pattern Recogn 43(5):1763–1775
Yin L, Basu A (2001) Generating realistic facial expressions with wrinkles for model-based coding. Comput Vis Image Underst 84(2):201–240
Yin L, Chen X, Sun Y, Worm T, Reale M (2008) A high-resolution 3D dynamic facial expression database. In: IEEE Proceedings on FG ’08. pp 1–6
Yin L, Wei X, Longo P, Bhuvanesh A (2006) Analyzing facial expressions using intensity-variant 3D data for human computer interaction. In: Proceedings on ICPR ’06. pp 1248–1251
Yin L, Wei X, Sun Y, Wang J, Rosato MJ (2006) A 3D facial expression database for facial behavior research. In: IEEE Proceedings on FGR ’06. pp 211–216
Zaharescu A, Boyer E, Varanasi K, Horaud R (2009) Surface feature detection and description with applications to mesh matching. In: Proceedings on IEEE computer society conference on computer vision and pattern recognition. pp 373–380
Zhang X, Reale M, Yin L (2013) Nebula feature: a space-time feature for posed and spontaneous 4D facial behavior analysis. In: IEEE FG ’13
Zhang X, Yin L, Cohn JF, Canavan S, Reale M, Horowitz A, Liu P (2013) A high-resolution spontaneous 3D dynamic facial expression database. In: IEEE FG ’13
Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans Pattern Anal Mach Intell 29(6):915–928
Acknowledgments
This research has been co-financed by the European Union (European Social Fund - ESF) and Greek national funds through the Operational Program “Education and Lifelong Learning” of the National Strategic Reference Framework (NSRF) - Research Funding Program: THALES-3DOR (MIS 379516).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Danelakis, A., Theoharis, T. & Pratikakis, I. A survey on facial expression recognition in 3D video sequences. Multimed Tools Appl 74, 5577–5615 (2015). https://doi.org/10.1007/s11042-014-1869-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-014-1869-6