Advertisement

The Visual Computer

, Volume 29, Issue 12, pp 1333–1350 | Cite as

Automatic facial expression recognition in real-time from dynamic sequences of 3D face scans

  • Stefano BerrettiEmail author
  • Alberto del Bimbo
  • Pietro Pala
Original Article

Abstract

In this paper, we present a fully-automatic and real-time approach for person-independent recognition of facial expressions from dynamic sequences of 3D face scans. In the proposed solution, first a set of 3D facial landmarks are automatically detected, then the local characteristics of the face in the neighborhoods of the facial landmarks and their mutual distances are used to model the facial deformation. Training two hidden Markov models for each facial expression to be recognized, and combining them to form a multiclass classifier, an average recognition rate of 79.4 % has been obtained for the 3D dynamic sequences showing the six prototypical facial expressions of the Binghamton University 4D Facial Expression database. Comparisons with competitor approaches on the same database show that our solution is able to obtain effective results with the advantage of being capable to process facial sequences in real-time.

Keywords

3D dynamic sequences 3D facial expression recognition Hidden Markov model Local descriptor 

Notes

Acknowledgements

The authors thank Professor Lijun Yin at Binghamton University for making available the BU-4DFE data set, and Marco Pompignoli, at the University of Firenze for writing part of the code for automatic detection of facial landmarks. A preliminary version of this work appeared in [7].

References

  1. 1.
    3dMD: http://www.3dmd.com (2010)
  2. 2.
    Asus: http://www.asus.com (2010)
  3. 3.
    Belongie, S., Malik, J., Puzicha, J.: Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell. 24(4), 509–521 (2002) CrossRefGoogle Scholar
  4. 4.
    Benedikt, L., Cosker, D., Rosin, P.L., Marshall, D.: Assessing the uniqueness and permanence of facial actions for use in biometric applications. IEEE Trans. Syst. Man Cybern., Part A, Syst. Hum. 40(3), 449–460 (2010) CrossRefGoogle Scholar
  5. 5.
    Berretti, S., del Bimbo, A., Pala, P.: 3D face recognition using iso-geodesic stripes. IEEE Trans. Pattern Anal. Mach. Intell. 32(12), 2162–2177 (2010) CrossRefGoogle Scholar
  6. 6.
    Berretti, S., Ben Amor, B., Daoudi, M., del Bimbo, A.: 3D facial expression recognition using SIFT descriptors of automatically detected keypoints. Vis. Comput. 27(11), 1021–1036 (2011) CrossRefGoogle Scholar
  7. 7.
    Berretti, S., del Bimbo, A., Pala, P.: Real-time expression recognition from dynamic sequences of 3D facial scans. In: Proc. 5th Eurographics/ACM SIGGRAPH Workshop on 3D Object Retrieval (3DOR’12), Cagliari, Italy, pp. 85–92 (2012) Google Scholar
  8. 8.
    Berretti, S., del Bimbo, A., Pala, P.: Superfaces: a super-resolution model for 3D faces. In: Proc. Workshop on Non-Rigid Shape Analysis and Deformable Image Alignment, Firenze, Italy, pp. 73–82 (2012) Google Scholar
  9. 9.
    Cootes, T., Edwards, G., Taylor, C.: Active appearance models. IEEE Trans. Pattern Anal. Mach. Intell. 23(6), 681–685 (2001) CrossRefGoogle Scholar
  10. 10.
    Creusot, C., Pears, N., Austin, J.: Automatic keypoint detection on 3D faces using a dictionary of local shapes. In: Proc. International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission, Hangzhou, China, pp. 204–211 (2011) CrossRefGoogle Scholar
  11. 11.
    Di3D: http://www.di3d.com (2006)
  12. 12.
    Drira, H., Ben Amor, B., Daoudi, M., Srivastava, A., Berretti, S.: 3D dynamic expression recognition based on a novel deformation vector field and random forest. In: Proc. International Conference on Pattern Recognition (ICPR’12), Tsukuba, Japan, pp. 1104–1107 (2012) Google Scholar
  13. 13.
    Ekman, P.: Universals and cultural differences in facial expressions of emotion. In: Proc. Nebraska Symposium on Motivation, Lincoln, NE, vol. 19, pp. 207–283 (1972) Google Scholar
  14. 14.
    Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978) Google Scholar
  15. 15.
    Fang, T., Zhao, X., Shah, S., Kakadiaris, I.: 4D facial expression recognition. In: Proc. IEEE International Conference on Computer Vision Workshop, Barcelona, Spain, pp. 1594–1601 (2011) Google Scholar
  16. 16.
    Fang, T., Zhao, X., Ocegueda, O., Shah, S.K., Kakadiaris, I.A.: 3D/4D facial expression analysis: an advanced annotated face model approach. Image Vis. Comput. 30(10), 738–749 (2012) CrossRefGoogle Scholar
  17. 17.
    Farkas, L.G., Munro, I.R.: Anthropometric Facial Proportions in Medicine. Thomas Books, Springfield (1987) Google Scholar
  18. 18.
    Fischler, M.A., Bolles, R.C.: Random sample consensus. Commun. ACM 24(6), 381–395 (1981) MathSciNetCrossRefGoogle Scholar
  19. 19.
    Frome, A., Huber, D., Kolluri, R., Bülow, T., Malik, J.: Recognizing objects in range data using regional point descriptors. In: Proc. European Conference on Computer Vision, Prague, Czech Republic, vol. 3, pp. 224–237 (2004) Google Scholar
  20. 20.
    Gupta, S., Markey, M.K., Bovik, A.C.: Anthropometric 3D face recognition. Int. J. Comput. Vis. 90(3), 331–349 (2010) CrossRefGoogle Scholar
  21. 21.
    Johnson, A., Hebert, M.: Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Trans. Pattern Anal. Mach. Intell. 21(5), 433–449 (1999) CrossRefGoogle Scholar
  22. 22.
    Kakadiaris, I.A., Passalis, G., Toderici, G., Murtuza, N., Lu, Y., Karampatziakis, N., Theoharis, T.: Three-dimensional face recognition in the presence of facial expressions: an annotated deformable approach. IEEE Trans. Pattern Anal. Mach. Intell. 29(4), 640–649 (2007) CrossRefGoogle Scholar
  23. 23.
    Kinect: http://www.xbox.com (2010)
  24. 24.
    Le, V., Tang, H., Huang, T.S.: Expression recognition from 3D dynamic faces using robust spatio-temporal shape features. In: Proc. IEEE Conference on Automatic Face and Gesture Recognition, Santa Barbara, CA, pp. 414–421 (2011) Google Scholar
  25. 25.
    Li, B., Mian, A., Liu, W., Krishna, A.: Using kinect for face recognition under varying poses, expressions, illumination and disguise. In: Proc. IEEE Workshop on the Applications of Computer Vision, Tampa, Florida, USA, pp. 186–192 (2013) Google Scholar
  26. 26.
    Linde, Y., Buzo, A., Gray, R.: An algorithm for vector quantizer design. IEEE Trans. Commun. 28(1), 84–94 (1980) CrossRefGoogle Scholar
  27. 27.
    Lowe, D.: Distinctive image features from scale-invariant key points. Int. J. Comput. Vis. 60(2), 91–110 (2004) CrossRefGoogle Scholar
  28. 28.
    Maalej, A., Ben Amor, B., Daoudi, M., Srivastava, A., Berretti, S.: Shape analysis of local facial patches for 3D facial expression recognition. Pattern Recognit. 44(8), 1581–1589 (2011) CrossRefGoogle Scholar
  29. 29.
    Matuszewski, B., Quan, W., Shark, L.K.: High-resolution comprehensive 3-D dynamic database for facial articulation analysis. In: Proc. IEEE International Conference on Computer Vision Workshops, Barcelona, Spain, pp. 2128–2135 (2011) Google Scholar
  30. 30.
    Matuszewski, B.J., Quan, W., Shark, L.K., McLoughlin, A.S., Lightbody, C.E., Emsley, H.C., Watkins, C.L.: Hi4d-adsip 3-D dynamic facial articulation database. Image Vis. Comput. 30(10), 713–727 (2012) CrossRefGoogle Scholar
  31. 31.
    Mehrabian, A., Wiener, M.: Decoding of inconsistent communications. J. Pers. Soc. Psychol. 6(1), 109–114 (1967) CrossRefGoogle Scholar
  32. 32.
    Mian, A.S., Bennamoun, M., Owens, R.: Keypoint detection and local feature matching for textured 3D face recognition. Int. J. Comput. Vis. 79(1), 1–12 (2008) CrossRefGoogle Scholar
  33. 33.
    Pandzic, I., Forchheimer, R.: MPEG-4 Facial Animation: The Standard, Implementation and Applications. Wiley, New York (2005) Google Scholar
  34. 34.
    Point grey: http://www.ptgrey.com (2010)
  35. 35.
    Rabiner, L.: A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77(2), 257–286 (1989) CrossRefGoogle Scholar
  36. 36.
    Rodriguez, J.J., Aggarwal, J.K.: Matching aerial images to 3-D terrain maps. IEEE Trans. Pattern Anal. Mach. Intell. 12(12), 1138–1149 (1990) CrossRefGoogle Scholar
  37. 37.
    Rusinkiewicz, S., Levoy, M.: Efficient variants of the ICP algorithm. In: Proc. International Conference on 3-D Digital Imaging and Modeling, Quebec City, Canada, pp. 145–152 (2001) CrossRefGoogle Scholar
  38. 38.
    Salazar, A., Wuhrer, S., Shu, C., Prieto, F.: Fully automatic expression-invariant face correspondence. Tech. Rep. arXiv:1202.1444v2 (2013). http://arxiv.org/abs/1202.1444v2
  39. 39.
    Samir, C., Srivastava, A., Daoudi, M., Klassen, E.: An intrinsic framework for analysis of facial surfaces. Int. J. Comput. Vis. 82(1), 80–95 (2009) CrossRefGoogle Scholar
  40. 40.
    Sandbach, G., Zafeiriou, S., Pantic, M., Rueckert, D.: A dynamic approach to the recognition of 3D facial expressions and their temporal models. In: Proc. IEEE Conference on Automatic Face and Gesture Recognition, Santa Barbara, CA, pp. 406–413 (2011) Google Scholar
  41. 41.
    Sandbach, G., Zafeiriou, S., Pantic, M., Rueckert, D.: Recognition of 3D facial expression dynamics. Image Vis. Comput. 30(10), 762–773 (2012) CrossRefGoogle Scholar
  42. 42.
    Sandbach, G., Zafeiriou, S., Pantic, M., Yin, L.: Static and dynamic 3D facial expression recognition: a comprehensive survey. Image Vis. Comput. 30(10), 683–697 (2012) CrossRefGoogle Scholar
  43. 43.
    Savran, A., Alyüz, N., Dibeklioǧlu, H., Çeliktutan, O., Gökberk, B., Sankur, B., Akarun, L.: Bosphorus database for 3D face analysis. In: Proc. First COST 2101 Workshop on Biometrics and Identity Management (2008) Google Scholar
  44. 44.
    Schneiderman, H.: Feature-centric evaluation for efficient cascaded object detection. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, pp. 29–36 (2004) Google Scholar
  45. 45.
    Schneiderman, H.: Learning a restricted Bayesian network for object detection. In: Proc. IEEE Conference on Computer Vision and Pattern Recognition, Washington, DC, USA, pp. 639–646 (2004) Google Scholar
  46. 46.
    Seol, Y., Seo, J., Kim, P.H., Lewis, J.P., Noh, J.: Weighted pose space editing for facial animation. Vis. Comput. 28(3), 319–327 (2012) CrossRefGoogle Scholar
  47. 47.
    Soyel, H., Demirel, H.: Facial Expression Recognition Using 3D Facial Feature Distances. InTech, Rijeka (2008) Google Scholar
  48. 48.
    Sun, Y., Yin, L.: Facial expression recognition based on 3D dynamic range model sequences. In: Proc. European Conference on Computer Vision, Marseille, France, pp. 58–71 (2008) Google Scholar
  49. 49.
    Sun, Y., Chen, X., Rosato, M., Yin, L.: Tracking vertex flow and model adaptation for 3D spatio-temporal face analysis. IEEE Trans. Syst. Man Cybern., Part A, Syst. Hum. 40(3), 461–474 (2010) CrossRefGoogle Scholar
  50. 50.
    Tang, H., Huang, T.S.: 3D facial expression recognition based on automatically selected features. In: Proc. IEEE International Conference on Computer Vision and Pattern Recognition, Anchorage, AK, pp. 1–8 (2008) Google Scholar
  51. 51.
    Tombari, F., Salti, S., Di Stefano, L.: Unique signature of histograms for local surface description. In: Proc. European Conference on Computer Vision, vol. III, Heraklion, Crete, Greece pp. 347–360 (2010) Google Scholar
  52. 52.
    Tsalakanidou, F., Malassiotis, S.: Real-time 2D+3D facial action and expression recognition. Pattern Recognit. 43(5), 1763–1775 (2010) CrossRefGoogle Scholar
  53. 53.
    Wang, Y., Liu, J., Tang, X.: Robust 3D face recognition by local shape difference boosting. IEEE Trans. Pattern Anal. Mach. Intell. 32(10), 1858–1870 (2010) CrossRefGoogle Scholar
  54. 54.
    Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.: A 3D facial expression database for facial behavior research. In: Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition, Southampton, UK, pp. 211–216 (2006) Google Scholar
  55. 55.
    Yin, L., Chen, X., Sun, Y., Worm, T., Reale, M.: A high-resolution 3D dynamic facial expression database. In: Proc. IEEE International Conference on Automatic Face and Gesture Recognition, Amsterdam, The Netherlands, pp. 1–6 (2008) Google Scholar
  56. 56.
    Zafeiriou, S., Yin, L.: 3D facial behaviour analysis and understanding. Image Vis. Comput. 30(10), 681–682 (2012) CrossRefGoogle Scholar
  57. 57.
    Zaharescu, A., Boyer, E., Varanasi, K., Horaud, R.: Surface feature detection and description with applications to mesh matching. In: Proc. IEEE International Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, pp. 373–380 (2009) Google Scholar
  58. 58.
    Zeng, Z., Pantic, M., Roisman, G., Huang, T.: A survey of affect recognition methods: audio, visual, and spontaneous expressions. IEEE Trans. Pattern Anal. Mach. Intell. 31(1), 39–58 (2009) CrossRefGoogle Scholar
  59. 59.
    Zhao, G., Pietikäinen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007) CrossRefGoogle Scholar
  60. 60.
    Zhao, X., Dellandréa, E., Chen, L., Samaras, D.: AU recognition on 3D faces based on an extended statistical facial feature model. In: Proc. IEEE International Conference on Biometrics: Theory, Applications and Systems, Washington, DC, USA, pp. 1–6 (2010) Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Stefano Berretti
    • 1
    Email author
  • Alberto del Bimbo
    • 1
  • Pietro Pala
    • 1
  1. 1.Dipartimento di Ingegneria dell’InformazioneUniversity of FirenzeFirenzeItaly

Personalised recommendations