The Visual Computer

, Volume 32, Issue 2, pp 257–269 | Cite as

A robust spatio-temporal scheme for dynamic 3D facial expression retrieval

  • Antonios Danelakis
  • Theoharis Theoharis
  • Ioannis Pratikakis
Original Article


The problem of facial expression recognition in dynamic sequences of 3D face scans has received a significant amount of attention in the recent past whereas the problem of retrieval in this type of data has not. A novel retrieval scheme for such data is introduced in this paper. It is the first spatio-temporal retrieval scheme ever used for retrieval in dynamic sequences of 3D face scans. The proposed scheme automatically detects specific facial landmarks and uses them to create a spatio-temporal descriptor. At first, geometric as well as topological information of the 3D face scans is captured by using the detected landmarks. In the sequel, the aforementioned spatial information is filtered by using wavelet transformation, resulting to our final spatio-temporal descriptor. Our descriptor is invariant to the number of the 3D face scans of a facial expression sequence. The proposed retrieval scheme exploits the Square of Euclidean distance in order to compare descriptors corresponding to different 3D facial sequences. A detailed evaluation of the introduced retrieval scheme is presented showing that it outperforms previous state-of-the-art retrieval schemes. Experiments have been conducted using the six prototypical expressions of the standard data set \(\textit{BU}-4\textit{DFE}\). Finally, a majority voting methodology based on the retrieval results is used to achieve unsupervised dynamic 3D facial expression recognition. The achieved classification accuracy outperforms the state-of-the-art supervised dynamic 3D facial expression recognition techniques.


Dynamic 3D mesh sequence 3D Object retrieval Facial expressions Wavelet transformation 



This research has been co-financed by the European Union (European Social Fund—ESF) and Greek national funds through the Operational Program “Education and Lifelong Learning” of the National Strategic Reference Framework (NSRF)—Research Funding Program: THALES-3DOR (MIS 379516).

In addition, special thanks should be dedicated to Takis Perakis, researcher of the Norwegian University of Science and Technology, for his precious help with the facial landmark detector.

Compliance with ethical standards

Conflict of Interest

The authors declare that they have no conflict of interest.


  1. 1.
    Berretti, S., Del Bimbo, A., Pala, P.: Automatic facial expression recognition in real-time from dynamic sequences of 3D face scans. Vis. Comput. 29(12), 1333–1350 (2013)CrossRefGoogle Scholar
  2. 2.
    Bovik, A.C.: Handbook of image and video processing (communications, networking and multimedia). Academic Press Inc., Orlando (2005)Google Scholar
  3. 3.
    Canavan, S.J., Sun, Y., Zhang, X., Yin, L.: A dynamic curvature based approach for facial activity analysis in 3D space. In: CVPR Workshops, pp. 14–19 (2012)Google Scholar
  4. 4.
    Danelakis, A., Theoharis, T., Pratikakis, I.: Geotopo: dynamic 3D facial expression retrieval using topological and geometric information. In: Proceedings of the 3D Object Retrieval 2014 Workshop, pp. 1–8 (2014)Google Scholar
  5. 5.
    Danelakis, A., Theoharis, T., Pratikakis, I.: A survey on facial expression recognition in 3D video sequences. Multimed. Tools. Appl. pp. 1–39 (2014)Google Scholar
  6. 6.
    Daubechies, I.: Ten lectures on wavelets. Society for Industrial and Applied Mathematics, Philadelphia (1992)CrossRefzbMATHGoogle Scholar
  7. 7.
    Drira, H., Ben Amor, B., Daoudi, M., Srivastava, A., Berretti, S.: 3D Dynamic expression recognition based on a novel deformation vector field and random forest. In: ICPR ’12, pp. 1104–1107 (2012)Google Scholar
  8. 8.
    Ekman, P., Friesen, W.: Facial action coding system: A technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  9. 9.
    Fang, T., Zhao, X., Ocegueda, O., Shah, S.K., Kakadiaris, I.A.: 3D/4D facial expression analysis: an advanced annotated face model approach. Image Vis. Comput. 30(10), 738–749 (2012)CrossRefGoogle Scholar
  10. 10.
    Fang, T., Zhao, X., Shah, S.K., Kakadiaris, I.A.: 4D Facial expression recognition. In: ICCV ’11, pp. 1594–1601 (2011)Google Scholar
  11. 11.
    Gebal, K., Bærentzen, J.A., Aanæs, H., Larsen, R.: Shape analysis using the auto diffusion function. In: Proceedings of the Symposium on Geometry Processing, SGP ’09, pp. 1405–1413 (2009)Google Scholar
  12. 12.
    Haar, F., Veltkamp, R.: 3D Face model fitting for recognition. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 652–664 (2008)Google Scholar
  13. 13.
    Jeni, L.A., Lórincz, A., Nagy, T., Palotai, Z., Sebók, J., Szabó, Z., Takács, D.: 3D Shape estimation in video sequences provides high precision evaluation of facial expressions. Image Vis. Comput. 30(10), 785–795 (2012)CrossRefGoogle Scholar
  14. 14.
    Matuszewski, B., Quan, W., Shark, L., McLoughlin, A., Lightbody, C., Emsley, H., Watkins, C.: Hi4D-ADSIP 3D dynamic facial articulation database. Elsevier Image Vis. Comput. 30(10), 713–727 (2012)CrossRefGoogle Scholar
  15. 15.
    Passalis, G., Theoharis, T., Kakadiaris, I.A.: PTK: a novel depth buffer-based shape descriptor for three-dimensional object retrieval. Vis. Comput. 23(1), 5–14 (2007)CrossRefGoogle Scholar
  16. 16.
    Perakis, P., Passalis, G., Theoharis, T., Kakadiaris, I.A.: 3D Facial landmark detection under large yaw and expression variations. IEEE Trans. Pattern Anal. Mach. Intell. 35(7), 1552–1564 (2013)CrossRefGoogle Scholar
  17. 17.
    Perakis, P., Theoharis, T., Kakadiaris, I.A.: Feature fusion for facial landmark detection. Pattern Recognit. 47(9), 2783–2793 (2014)CrossRefGoogle Scholar
  18. 18.
    Quiroga, R.Q., Sakowitz, O.W., Basar, E., Schürmann, M.: Wavelet transform in the analysis of the frequency composition of evoked potentials. Brain Res. Protoc. 8(1), 16–24 (2001)CrossRefGoogle Scholar
  19. 19.
    Sandbach, G., Zafeiriou, S., Pantic, M., Rueckert, D.: Recognition of 3D facial expression dynamics. Elsevier Image Vis. Comput. 30(10), 762–773 (2012)CrossRefGoogle Scholar
  20. 20.
    Sfikas, K., Theoharis, T., Pratikakis, I.: Non-rigid 3D object retrieval using topological information guided by conformal factors. Vis. Comput. 28(9), 943–955 (2012)CrossRefGoogle Scholar
  21. 21.
    Sfikas, K., Theoharis, T., Pratikakis, I.: 3D Object retrieval via range image queries in a bag-of-visual-words context. Vis. Comput. 29(12), 1351–1361 (2013)CrossRefGoogle Scholar
  22. 22.
    Sun, J., Ovsjanikov, M., Guibas, L.: A concise and provably informative multi-scale signature based on heat diffusion. In: Proceedings of the Symposium on Geometry Processing, SGP ’09, pp. 1383–1392. Eurographics Association (2009)Google Scholar
  23. 23.
    Sun, Y., Chen, X., Rosato, M.J., Yin, L.: Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis. IEEE Trans. Syst. Man Cybern. Part A 40(3), 461–474 (2010)CrossRefGoogle Scholar
  24. 24.
    Sun, Y., Reale, M., Yin, L.: Recognizing partial facial action units based on 3D dynamic range data for facial expression recognition. In: FG ’08, pp. 1–8 (2008)Google Scholar
  25. 25.
    Sun, Y., Yin, L.: Facial expression recognition based on 3D dynamic range model sequences. In: Springer Proceedings of the ECCV ’08: Part II, pp. 58–71 (2008)Google Scholar
  26. 26.
    Tsalakanidou, F., Malassiotis, S.: Robust facial action recognition from real-time 3D streams. In: CVPR ’09, pp. 4–11 (2009)Google Scholar
  27. 27.
    Tsalakanidou, F., Malassiotis, S.: Real-time 2D + 3D facial action and expression recognition. Elsevier Pattern Recognit. 43(5), 1763–1775 (2010)CrossRefGoogle Scholar
  28. 28.
    Yin, L., Chen, X., Sun, Y., Worm, T., Reale, M.: A high-resolution 3D dynamic facial expression database. In: IEEE Proceedings of the FG ’08, pp. 1–6 (2008)Google Scholar
  29. 29.
    Yin, L., Wei, X., Longo, P., Bhuvanesh, A.: Analyzing facial expressions using intensity-variant 3D data for human computer interaction. In: Proceedings of the ICPR ’06, pp. 1248–1251 (2006)Google Scholar
  30. 30.
    Zhang, X., Reale, M., Yin, L.: Nebula feature: A space-time feature for posed and spontaneous 4D facial behavior analysis. In: IEEE FG ’13 (2013)Google Scholar
  31. 31.
    Zhang, X., Yin, L., Cohn, J.F., Canavan, S., Reale, M., Horowitz, A., Liu, P.: A high-resolution spontaneous 3D dynamic facial expression database. In: IEEE FG ’13 (2013)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2015

Authors and Affiliations

  • Antonios Danelakis
    • 1
  • Theoharis Theoharis
    • 1
    • 2
  • Ioannis Pratikakis
    • 3
  1. 1.Department of Informatics and TelecommunicationsUniversity of AthensAthensGreece
  2. 2.Department of Computer and Information ScienceNorwegian University of Science and TechnologyTrondheimNorway
  3. 3.Department of Electrical and Computer EngineeringDemocritus University of ThraceXanthiGreece

Personalised recommendations