Realistic Face Animation for Audiovisual Speech Applications: A Densification Approach Driven by Sparse Stereo Meshes

  • Marie-Odile Berger
  • Jonathan Ponroy
  • Brigitte Wrobel-Dautcourt
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5496)


Being able to produce realistic facial animation is crucial for many speech applications in language learning technologies. Reaching realism needs to acquire and to animate dense 3D models of the face which are often acquired with 3D scanners. However, acquiring the dynamics of the speech from 3D scans is difficult as the acquisition time generally allows only sustained sounds to be recorded. On the contrary, acquiring the speech dynamics on a sparse set of points is easy using a stereovision recording a talker with markers painted on his/her face. In this paper, we propose an approach to animate a very realistic dense talking head which makes use of a reduced set of 3D dense meshes acquired for sustained sounds as well as the speech dynamics learned on a talker painted with white markers. The contributions of the paper are twofold: We first propose an appropriate principal component analysis (PCA) with missing data techniques in order to compute the basic modes of the speech dynamics despite possible unobservable points in the sparse meshes obtained by the stereovision system. We then propose a method for densifying the modes, that is a method for computing the dense modes for spatial animation from the sparse modes learned by the stereovision system. Examples prove the effectiveness of the approach and the high realism obtained with our method.


Face animation densification PCA with missing data 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Berger, M.O.: Realistic face animation from sparse stereo meshes. In: International Conference on Auditory-Visual Speech Processing 2007 (2007)Google Scholar
  2. 2.
    Blanz, V., Vetter, T.: A morphable model for the synthesis of 3d faces. In: SIGGRAPH 1999, pp. 187–194 (1999)Google Scholar
  3. 3.
    Ezzat, T., Geiger, G., Poggio, T.: Trainable videorealistic speech animation. In: Proceedings of the 29th annual conference on Computer graphics and interactive techniques, pp. 388–398. ACM Press, New York (2002)Google Scholar
  4. 4.
    Kalberer, G., Van Gool, L.: Face animation based on observed 3d speech dynamics. In: Proceedings of Computer Animation 2001 Conference, pp. 20–27 (2001)Google Scholar
  5. 5.
    Kuratate, T.: Talking Head Animation System Driven by Facial Motion Mapping and a 3D face Database. PhD thesis, Nara Institute of Science and Technology (2004)Google Scholar
  6. 6.
    Muller, P., Kalberer, G., Proesmans, M., Van Gool, L.: Realistic speech animation based on observed 3d face dynamics. Vision Image and Signal Processing 152(4) (2005)Google Scholar
  7. 7.
    Munhall, K., Vatikiotis-Bateson, E.: The moving face during speech communication. In: Hearing by Eyes, vol. 2, ch. 6, pp. 123–139. Psychology Press, San Diego (1998)Google Scholar
  8. 8.
    Roweis, S.: Em algorithms for pca and spca. In: Advances in Neural Information Processing Systems, pp. 626–632. IEEE and ACM (1998)Google Scholar
  9. 9.
    Skočaj, D., Bischof, H., Leonardis, A.: A robust PCA algorithm for building representations from panoramic images. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2353, pp. 761–775. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  10. 10.
    Summer, R., Popovic, J.: Deformable transfer for triangle meshes. In: Procceding of SIGGRAPH (2004)Google Scholar
  11. 11.
    Wrobel-Dautcourt, B., Berger, M.-O., Potard, B., Laprie, Y., Ouni, S.: A low-cost stereovision based system for acquisition of visible articulatory data. In: Proceedings of the 5th Conference on Auditory-Visual Speech Processing (AVSP), Vancouver Island, BC, Canada, July 2005, pp. 65–70 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Marie-Odile Berger
    • 1
  • Jonathan Ponroy
    • 1
  • Brigitte Wrobel-Dautcourt
    • 1
  1. 1.LORIA/INRIA Nancy Grand EstFrance

Personalised recommendations