Skip to main content
View expanded cover

International Conference on Intelligent Data Engineering and Automated Learning

IDEAL 2002: Intelligent Data Engineering and Automated Learning — IDEAL 2002 pp 557–561Cite as

Viseme Recognition Experiment Using Context Dependent Hidden Markov Models

Part of the Lecture Notes in Computer Science book series (LNCS,volume 2412)

Abstract

Visual images synchronized with audio signals can provide user-friendly interface for man machine interactions. The visual speech can be represented as a sequence of visemes, which are the generic face images corresponding to particular sounds. We use HMMs (hidden Markov models) to convert audio signals to a sequence of visemes. In this paper, we compare two approaches in using HMMs. In the first approach, an HMM is trained for each triviseme which is a viseme with its left and right context, and the audio signals are directly recognized as a sequence of trivisemes. In the second approach, each triphone is modeled with an HMM, and a general triphone recognizer is used to produce a triphone sequence from the audio signals. The triviseme or triphone sequence is then converted to a viseme sequence. The performances of the two viseme recognition systems are evaluated on the TIMIT speech corpus.

Keywords

  • Hide Markov Model
  • Speech Recognition
  • Facial Image
  • Audio Signal
  • Automatic Speech Recognition

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/3-540-45675-9_84
  • Chapter length: 5 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   119.00
Price excludes VAT (USA)
  • ISBN: 978-3-540-45675-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   159.00
Price excludes VAT (USA)
Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Choi, K., Hwang, J.: Baum-Welch HMM inversion for audio-to-visual conversion, IEEE International Workshop on Multimedia Signal Processing, pp. 175–180, 1999.

    Google Scholar 

  2. Fisher, C.: Confusions among visually perceived consonants, Journal on Speech and Hearing Research, vol. 11, pp. 796–804, 1968.

    Google Scholar 

  3. Grant, K., Walden, B., Seitz, P.: Auditory-visual speech recognition by hearing-impaired subjects: consonant recognition, sentence recognition, and auditory-visual integration, Journal of Acoustic Society of America, vol. 103, pp. 2677–2690, 1998.

    CrossRef  Google Scholar 

  4. Morishima, S., Harashima, H.: A media conversion from speech to facial image for intelligent man-machine interface, IEEE Journal on selected areas in communications, vol. 9, no. 4, pp. 594–600, 1991.

    CrossRef  Google Scholar 

  5. Rabiner, L.: A tutorial on hidden Markov models and selected applications in speech recognition, Proceedings of the IEEE, vol. 77, no. 2, pp. 257–286, 1989.

    CrossRef  Google Scholar 

  6. Rao, R., Chen, T.: Mersereau, R., Audio-to-visual conversion for multimedia communication, IEEE Transaction on Industrial Electronics, vol. 45, no. 1, pp. 15–22, 1998.

    CrossRef  Google Scholar 

  7. Rogozan, A., Delelise, P.: Adaptive fusion of acoustic and visual sources for automatic speech recognition, Speech Communication, vol. 26, pp. 149–161, 1998.

    CrossRef  Google Scholar 

  8. Tamura, S., Waibel, A.: Noise reduction using connectionist models, IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 553–556, 1988.

    Google Scholar 

  9. TIMIT: Acoustic-phonetic continuous speech corpus, Nist Speech Disc 1-1.1, October 1990.

    Google Scholar 

  10. Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., Lang, K.: Phoneme recognition using time-delay neural networks, IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 37, no. 3, pp. 328–339, 1989.

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Speech Information Processing Laboratory Department of Computer Science and Engineering, Korea University, Sungbookgoo Anamdong 5-1, Seoul, Korea, 136-701

    Soonkyu Lee & Dongsuk Yook

Authors
  1. Soonkyu Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dongsuk Yook
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Electrical Engineering and Electronics, UMIST, Manchester, M60 1QD, UK

    Hujun Yin, Nigel Allinson & Richard Freeman,  & 

  2. Department of Computation, UMIST, Manchester, M60 1QD, UK

    John Keane

  3. Department of Biomolecular Science, UMIST, Manchester, M60 1QD, UK

    Simon Hubbard

Rights and permissions

Reprints and Permissions

Copyright information

© 2002 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Lee, S., Yook, D. (2002). Viseme Recognition Experiment Using Context Dependent Hidden Markov Models. In: Yin, H., Allinson, N., Freeman, R., Keane, J., Hubbard, S. (eds) Intelligent Data Engineering and Automated Learning — IDEAL 2002. IDEAL 2002. Lecture Notes in Computer Science, vol 2412. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45675-9_84

Download citation

  • DOI: https://doi.org/10.1007/3-540-45675-9_84

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-44025-3

  • Online ISBN: 978-3-540-45675-9

  • eBook Packages: Springer Book Archive