Skip to main content
SpringerLink
Log in

Automatic facial expression recognition in real-time from dynamic sequences of 3D face scans

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In this paper, we present a fully-automatic and real-time approach for person-independent recognition of facial expressions from dynamic sequences of 3D face scans. In the proposed solution, first a set of 3D facial landmarks are automatically detected, then the local characteristics of the face in the neighborhoods of the facial landmarks and their mutual distances are used to model the facial deformation. Training two hidden Markov models for each facial expression to be recognized, and combining them to form a multiclass classifier, an average recognition rate of 79.4 % has been obtained for the 3D dynamic sequences showing the six prototypical facial expressions of the Binghamton University 4D Facial Expression database. Comparisons with competitor approaches on the same database show that our solution is able to obtain effective results with the advantage of being capable to process facial sequences in real-time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. 3dMD: http://www.3dmd.com (2010)

  2. Asus: http://www.asus.com (2010)

  3. Belongie, S., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell. 24(4), 509–521 (2002)

    Article  Google Scholar 

  4. Benedikt, L., Cosker, D., Rosin, P.L., Marshall, D.: Assessing the uniqueness and permanence of facial actions for use in biometric applications. IEEE Trans. Syst. Man Cybern., Part A, Syst. Hum. 40(3), 449–460 (2010)

    Article  Google Scholar 

  5. Berretti, S., del Bimbo, A., Pala, P.: 3D face recognition using iso-geodesic stripes. IEEE Trans. Pattern Anal. Mach. Intell. 32(12), 2162–2177 (2010)

    Article  Google Scholar 

  6. Berretti, S., Ben Amor, B., Daoudi, M., del Bimbo, A.: 3D facial expression recognition using SIFT descriptors of automatically detected keypoints. Vis. Comput. 27(11), 1021–1036 (2011)

    Article  Google Scholar 

  7. Berretti, S., del Bimbo, A., Pala, P.: Real-time expression recognition from dynamic sequences of 3D facial scans. In: Proc. 5th Eurographics/ACM SIGGRAPH Workshop on 3D Object Retrieval (3DOR’12), Cagliari, Italy, pp. 85–92 (2012)

    Google Scholar 

  8. Berretti, S., del Bimbo, A., Pala, P.: Superfaces: a super-resolution model for 3D faces. In: Proc. Workshop on Non-Rigid Shape Analysis and Deformable Image Alignment, Firenze, Italy, pp. 73–82 (2012)

    Google Scholar 

  9. Cootes, T., Edwards, G., Taylor, C.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001)

    Article  Google Scholar 

  10. Creusot, C., Pears, N., Austin, J.: Automatic keypoint detection on 3D faces using a dictionary of local shapes. In: Proc. International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission, Hangzhou, China, pp. 204–211 (2011)

    Chapter  Google Scholar 

  11. Di3D: http://www.di3d.com (2006)

  12. Drira, H., Ben Amor, B., Daoudi, M., Srivastava, A., Berretti, S.: 3D dynamic expression recognition based on a novel deformation vector field and random forest. In: Proc. International Conference on Pattern Recognition (ICPR’12), Tsukuba, Japan, pp. 1104–1107 (2012)

    Google Scholar 

  13. Ekman, P.: Universals and cultural differences in facial expressions of emotion. In: Proc. Nebraska Symposium on Motivation, Lincoln, NE, vol. 19, pp. 207–283 (1972)

    Google Scholar 

  14. Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)

    Google Scholar 

  15. Fang, T., Zhao, X., Shah, S., Kakadiaris, I.: 4D facial expression recognition. In: Proc. IEEE International Conference on Computer Vision Workshop, Barcelona, Spain, pp. 1594–1601 (2011)

    Google Scholar 

  16. Fang, T., Zhao, X., Ocegueda, O., Shah, S.K., Kakadiaris, I.A.: 3D/4D facial expression analysis: an advanced annotated face model approach. Image Vis. Comput. 30(10), 738–749 (2012)

    Article  Google Scholar 

  17. Farkas, L.G., Munro, I.R.: Anthropometric Facial Proportions in Medicine. Thomas Books, Springfield (1987)

    Google Scholar 

  18. Fischler, M.A., Bolles, R.C.: Random sample consensus. Commun. ACM 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  19. Frome, A., Huber, D., Kolluri, R., Bülow, T., Malik, J.: Recognizing objects in range data using regional point descriptors. In: Proc. European Conference on Computer Vision, Prague, Czech Republic, vol. 3, pp. 224–237 (2004)

    Google Scholar 

  20. Gupta, S., Markey, M.K., Bovik, A.C.: Anthropometric 3D face recognition. Int. J. Comput. Vis. 90(3), 331–349 (2010)

    Article  Google Scholar 

  21. Johnson, A., Hebert, M.: Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Trans. Pattern Anal. Mach. Intell. 21(5), 433–449 (1999)

    Article  Google Scholar 

  22. Kakadiaris, I.A., Passalis, G., Toderici, G., Murtuza, N., Lu, Y., Karampatziakis, N., Theoharis, T.: Three-dimensional face recognition in the presence of facial expressions: an annotated deformable approach. IEEE Trans. Pattern Anal. Mach. Intell. 29(4), 640–649 (2007)

    Article  Google Scholar 

  23. Kinect: http://www.xbox.com (2010)

  24. Le, V., Tang, H., Huang, T.S.: Expression recognition from 3D dynamic faces using robust spatio-temporal shape features. In: Proc. IEEE Conference on Automatic Face and Gesture Recognition, Santa Barbara, CA, pp. 414–421 (2011)

    Google Scholar 

  25. Li, B., Mian, A., Liu, W., Krishna, A.: Using kinect for face recognition under varying poses, expressions, illumination and disguise. In: Proc. IEEE Workshop on the Applications of Computer Vision, Tampa, Florida, USA, pp. 186–192 (2013)

    Google Scholar 

  26. Linde, Y., Buzo, A., Gray, R.: An algorithm for vector quantizer design. IEEE Trans. Commun. 28(1), 84–94 (1980)

    Article  Google Scholar 

  27. Lowe, D.: Distinctive image features from scale-invariant key points. Int. J. Comput. Vis. 60(2), 91–110 (2004)

    Article  Google Scholar 

  28. Maalej, A., Ben Amor, B., Daoudi, M., Srivastava, A., Berretti, S.: Shape analysis of local facial patches for 3D facial expression recognition. Pattern Recognit. 44(8), 1581–1589 (2011)

    Article  Google Scholar 

  29. Matuszewski, B., Quan, W., Shark, L.K.: High-resolution comprehensive 3-D dynamic database for facial articulation analysis. In: Proc. IEEE International Conference on Computer Vision Workshops, Barcelona, Spain, pp. 2128–2135 (2011)

    Google Scholar 

  30. Matuszewski, B.J., Quan, W., Shark, L.K., McLoughlin, A.S., Lightbody, C.E., Emsley, H.C., Watkins, C.L.: Hi4d-adsip 3-D dynamic facial articulation database. Image Vis. Comput. 30(10), 713–727 (2012)

    Article  Google Scholar 

  31. Mehrabian, A., Wiener, M.: Decoding of inconsistent communications. J. Pers. Soc. Psychol. 6(1), 109–114 (1967)

    Article  Google Scholar 

  32. Mian, A.S., Bennamoun, M., Owens, R.: Keypoint detection and local feature matching for textured 3D face recognition. Int. J. Comput. Vis. 79(1), 1–12 (2008)

    Article  Google Scholar 

  33. Pandzic, I., Forchheimer, R.: MPEG-4 Facial Animation: The Standard, Implementation and Applications. Wiley, New York (2005)

    Google Scholar 

  34. Point grey: http://www.ptgrey.com (2010)

  35. Rabiner, L.: A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77(2), 257–286 (1989)

    Article  Google Scholar 

  36. Rodriguez, J.J., Aggarwal, J.K.: Matching aerial images to 3-D terrain maps. IEEE Trans. Pattern Anal. Mach. Intell. 12(12), 1138–1149 (1990)

    Article  Google Scholar 

  37. Rusinkiewicz, S., Levoy, M.: Efficient variants of the ICP algorithm. In: Proc. International Conference on 3-D Digital Imaging and Modeling, Quebec City, Canada, pp. 145–152 (2001)

    Chapter  Google Scholar 

  38. Salazar, A., Wuhrer, S., Shu, C., Prieto, F.: Fully automatic expression-invariant face correspondence. Tech. Rep. arXiv:1202.1444v2 (2013). http://arxiv.org/abs/1202.1444v2

  39. Samir, C., Srivastava, A., Daoudi, M., Klassen, E.: An intrinsic framework for analysis of facial surfaces. Int. J. Comput. Vis. 82(1), 80–95 (2009)

    Article  Google Scholar 

  40. Sandbach, G., Zafeiriou, S., Pantic, M., Rueckert, D.: A dynamic approach to the recognition of 3D facial expressions and their temporal models. In: Proc. IEEE Conference on Automatic Face and Gesture Recognition, Santa Barbara, CA, pp. 406–413 (2011)

    Google Scholar 

  41. Sandbach, G., Zafeiriou, S., Pantic, M., Rueckert, D.: Recognition of 3D facial expression dynamics. Image Vis. Comput. 30(10), 762–773 (2012)

    Article  Google Scholar 

  42. Sandbach, G., Zafeiriou, S., Pantic, M., Yin, L.: Static and dynamic 3D facial expression recognition: a comprehensive survey. Image Vis. Comput. 30(10), 683–697 (2012)

    Article  Google Scholar 

  43. Savran, A., Alyüz, N., Dibeklioǧlu, H., Çeliktutan, O., Gökberk, B., Sankur, B., Akarun, L.: Bosphorus database for 3D face analysis. In: Proc. First COST 2101 Workshop on Biometrics and Identity Management (2008)

    Google Scholar 

  44. Schneiderman, H.: Feature-centric evaluation for efficient cascaded object detection. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, pp. 29–36 (2004)

    Google Scholar 

  45. Schneiderman, H.: Learning a restricted Bayesian network for object detection. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, pp. 639–646 (2004)

    Google Scholar 

  46. Seol, Y., Seo, J., Kim, P.H., Lewis, J.P., Noh, J.: Weighted pose space editing for facial animation. Vis. Comput. 28(3), 319–327 (2012)

    Article  Google Scholar 

  47. Soyel, H., Demirel, H.: Facial Expression Recognition Using 3D Facial Feature Distances. InTech, Rijeka (2008)

    Google Scholar 

  48. Sun, Y., Yin, L.: Facial expression recognition based on 3D dynamic range model sequences. In: Proc. European Conference on Computer Vision, Marseille, France, pp. 58–71 (2008)

    Google Scholar 

  49. Sun, Y., Chen, X., Rosato, M., Yin, L.: Tracking vertex flow and model adaptation for 3D spatio-temporal face analysis. IEEE Trans. Syst. Man Cybern., Part A, Syst. Hum. 40(3), 461–474 (2010)

    Article  Google Scholar 

  50. Tang, H., Huang, T.S.: 3D facial expression recognition based on automatically selected features. In: Proc. IEEE International Conference on Computer Vision and Pattern Recognition, Anchorage, AK, pp. 1–8 (2008)

    Google Scholar 

  51. Tombari, F., Salti, S., Di Stefano, L.: Unique signature of histograms for local surface description. In: Proc. European Conference on Computer Vision, vol. III, Heraklion, Crete, Greece pp. 347–360 (2010)

    Google Scholar 

  52. Tsalakanidou, F., Malassiotis, S.: Real-time 2D+3D facial action and expression recognition. Pattern Recognit. 43(5), 1763–1775 (2010)

    Article  Google Scholar 

  53. Wang, Y., Liu, J., Tang, X.: Robust 3D face recognition by local shape difference boosting. IEEE Trans. Pattern Anal. Mach. Intell. 32(10), 1858–1870 (2010)

    Article  Google Scholar 

  54. Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.: A 3D facial expression database for facial behavior research. In: Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition, Southampton, UK, pp. 211–216 (2006)

    Google Scholar 

  55. Yin, L., Chen, X., Sun, Y., Worm, T., Reale, M.: A high-resolution 3D dynamic facial expression database. In: Proc. IEEE International Conference on Automatic Face and Gesture Recognition, Amsterdam, The Netherlands, pp. 1–6 (2008)

    Google Scholar 

  56. Zafeiriou, S., Yin, L.: 3D facial behaviour analysis and understanding. Image Vis. Comput. 30(10), 681–682 (2012)

    Article  Google Scholar 

  57. Zaharescu, A., Boyer, E., Varanasi, K., Horaud, R.: Surface feature detection and description with applications to mesh matching. In: Proc. IEEE International Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, pp. 373–380 (2009)

    Google Scholar 

  58. Zeng, Z., Pantic, M., Roisman, G., Huang, T.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31(1), 39–58 (2009)

    Article  Google Scholar 

  59. Zhao, G., Pietikäinen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

  60. Zhao, X., Dellandréa, E., Chen, L., Samaras, D.: AU recognition on 3D faces based on an extended statistical facial feature model. In: Proc. IEEE International Conference on Biometrics: Theory, Applications and Systems, Washington, DC, USA, pp. 1–6 (2010)

    Google Scholar 

Download references

Acknowledgements

The authors thank Professor Lijun Yin at Binghamton University for making available the BU-4DFE data set, and Marco Pompignoli, at the University of Firenze for writing part of the code for automatic detection of facial landmarks. A preliminary version of this work appeared in [7].

Author information

Authors and Affiliations

  1. Dipartimento di Ingegneria dell’Informazione, University of Firenze, Firenze, Italy

    Stefano Berretti, Alberto del Bimbo & Pietro Pala

Authors
  1. Stefano Berretti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alberto del Bimbo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pietro Pala
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stefano Berretti.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Berretti, S., del Bimbo, A. & Pala, P. Automatic facial expression recognition in real-time from dynamic sequences of 3D face scans. Vis Comput 29, 1333–1350 (2013). https://doi.org/10.1007/s00371-013-0869-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-013-0869-2

Keywords

Search

Navigation