Advertisement

A Proposal for a Visual Speech Animation System for European Portuguese

  • José Serra
  • Manuel Ribeiro
  • João Freitas
  • Verónica Orvalho
  • Miguel Sales Dias
Part of the Communications in Computer and Information Science book series (CCIS, volume 328)

Abstract

Visual speech animation, or lip synchronization, is the process of matching speech with the lip movements of a virtual character. It is a challenging task because all articulatory movements must be controlled and synchronized with the audio signal. Existing language-independent systems usually require fine tuning by an artist to avoid artefacts appearing in the animation. In this paper, we present a modular visual speech animation framework aimed at speeding up and easing the visual speech animation process as compared with traditional techniques. We demonstrate the potential of the framework by developing the first automatic visual speech automation system for European Portuguese based on the concatenation of visemes. We also present the results of a preliminary evaluation that was carried out to assess the quality of two different phoneme-to-viseme mappings devised for the language.

Keywords

visual speech animation phoneme-to-viseme mapping European Portuguese virtual characters 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Ostendorf, M.: Moving beyond the ‘beads-on-a-string’ model of speech. In: Proceedings of IEEE ASRU 1999, Keystone, CO, USA (December 1999)Google Scholar
  2. 2.
    Cohen, M., Massaro, D.: Modeling coarticulation in synthetic visual speech. Models and Techniques in Computer, 139–156 (1993)Google Scholar
  3. 3.
    Bregler, C., Covell, M., Slaney, M.: Video Rewrite. In: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1997, New York, USA, pp. 353–360 (1997)Google Scholar
  4. 4.
    Zhou, Z., Zhao, G., Pietikäinen, M.: Synthesizing a talking mouth. In: Proc. of the 7th Indian Conf. on Computer Vision, Graphics and Image Processing, ICVGIP 2010, New York, USA, pp. 211–218 (2010)Google Scholar
  5. 5.
    Liu, K., Ostermann, J.: Optimization of an Image-Based Talking Head System. EURASIP Journal on Audio, Speech, and Music Processing, 1–13 (2009)Google Scholar
  6. 6.
    Gutierrez-Osuna, R., Kakumanu, P., Esposito, A., Garcia, O., Bojorquez, A., Castillo, J., Rudomin, I.: Speech-driven facial animation with realistic dynamics. IEEE Transactions on Multimedia 7(1), 33–42 (2005)CrossRefGoogle Scholar
  7. 7.
    Liu, J., You, M., Chen, C., Song, M.: Real-time speechdriven animation of expressive talking faces. International Journal of General Systems 40(4), 439–455 (2009)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Hofer, G., Yamagishi, J., Shimodaira, H.: Speech-driven Lip Motion Generation with a Trajectory HMM. In: Proc. of Interspeech, pp. 2314–2317 (2008)Google Scholar
  9. 9.
    Demartino, J., Pinimagalhaes, L., Violaro, F.: Facial Animation based on context-dependent visemes. Computers & Graphics 30(6) (2006)Google Scholar
  10. 10.
    Berger, M., Hofer, G.: Carnival - Combining Speech Technology and Computer Animation. IEEE Computer Graphics and Applications 31, 80–89 (2011)CrossRefGoogle Scholar
  11. 11.
    Sutton, S., Cole, R., Villiers, J., Schalkwyk, J., Vermeulen, P., Macon, M., Yan, Y., Kaiser, E., Rundle, B., Shobaki, K., Hosom, P., Kain, A., Wouters, J., Massaro, D., Cohen, M.: Universal Speech Tools: The CSLY toolkit. Language, 3221–3224 (1998)Google Scholar
  12. 12.
    Erber, N.: Auditory, Visual, and Auditory-visual Recognition of Consonants by Children with Normal and Impaired Hearing. Journal of Speech and Hearing Research 15, 413–422 (1972)Google Scholar
  13. 13.
  14. 14.
    Verwey, J., Blake, E.: The Influence of Lip Animation on the Perception of Speech in Virtual Environments. In: Proc. of the 8th Annual International Workshop on Presence, University College London, pp. 163–170 (2005)Google Scholar
  15. 15.
    Autodesk Maya (March 30, 2012), http://usa.autodesk.com/maya/

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • José Serra
    • 1
    • 2
  • Manuel Ribeiro
    • 3
    • 4
  • João Freitas
    • 3
    • 4
  • Verónica Orvalho
    • 1
    • 2
  • Miguel Sales Dias
    • 3
    • 4
  1. 1.Instituto de TelecomunicaçõesPortoPortugal
  2. 2.Department of Computer Science, Faculty of ScienceUniversity of PortoPortoPortugal
  3. 3.Microsoft Language Development CenterLisbonPortugal
  4. 4.ISCTE-Lisbon University Institute/ADETTI-IULLisboaPortugal

Personalised recommendations