Skip to main content

Audio-to-Visual Conversion Using Hidden Markov Models

Part of the Lecture Notes in Computer Science book series (LNAI,volume 2417)

Abstract

We describe audio-to-visual conversion techniques for efficient multimedia communications. The audio signals are automatically converted to visual images of mouth shape. The visual speech can be represented as a sequence of visemes, which are the generic face images corresponding to particular sounds. Visual images synchronized with audio signals can provide user-friendly interface for man machine interactions. Also, it can be used to help the people with impaired-hearing. We use HMMs (hidden Markov models) to convert audio signals to a sequence of visemes. In this paper, we compare two approaches in using HMMs. In the first approach, an HMM is trained for each viseme, and the audio signals are directly recognized as a sequence of visemes. In the second approach, each phoneme is modeled with an HMM, and a general phoneme recognizer is utilized to produce a phoneme sequence from the audio signals. The phoneme sequence is then converted to a viseme sequence. We implemented the two approaches and tested them on the TIMIT speech corpus. The viseme recognizer shows 33.9% error rate, and the phoneme-based approach exhibits 29.7% viseme recognition error rate. When similar viseme classes are merged, we have found that the error rates can be reduced to 20.5% and 13.9%, respectably.

Keywords

  • Hide Markov Model
  • Facial Image
  • Audio Signal
  • Visual Speech
  • Conversion Technique

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/3-540-45683-X_60
  • Chapter length: 8 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   119.00
Price excludes VAT (USA)
  • ISBN: 978-3-540-45683-4
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   159.00
Price excludes VAT (USA)

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. K. Choi, and J. Hwang, “Baum-Welch HMM inversion for audio-to-visual conversion”, IEEE International Workshop on Multimedia Signal Processing, pp. 175–180, 1999.

    Google Scholar 

  2. C. Fisher, “Confusions among visually perceived consonants”, Journal on Speech and Hearing Research, vol. 11, pp. 796–804, 1968.

    Google Scholar 

  3. K. Grant, B. Walden, and P. Seitz, “Auditory-visual speech recognition by hearing-impaired subjects: consonant recognition, sentence recognition, and auditory-visual integration”, Journal of Acoustic Society of America, vol. 103, pp. 2677–2690, 1998.

    CrossRef  Google Scholar 

  4. S. Morishima and H. Harashima, “A media conversion from speech to facial image for intelligent man-machine interface”, IEEE Journal on selected areas in communications, vol. 9, no. 4, pp. 594–600, 1991.

    CrossRef  Google Scholar 

  5. Nist Speech Disc 1-1.1, TIMIT Acoustic-phonetic continuous speech corpus, October 1990.

    Google Scholar 

  6. L. Rabiner, “A tutorial on hidden Markov models and selected applications in speech recognition”, Proceedings of the IEEE, vol. 77, no.2, pp. 257–286, 1989.

    CrossRef  Google Scholar 

  7. R. Rao, T. Chen and R. Mersereau, “Audio-to-visual conversion for multimedia communication”, IEEE Transaction on Industrial Electronics, vol. 45, no. 1, pp. 15–22, 1998.

    CrossRef  Google Scholar 

  8. A. Rogozan and P. Delelise, “Adaptive fusion of acoustic and visual sources for automatic speech recognition”, Speech Communication, vol. 26, pp. 149–161, 1998.

    CrossRef  Google Scholar 

  9. S. Tamura and A. Waibel, “Noise reduction using connectionist models”, IEEE International Conference on Acoustics, Speech, and Signal Processing, pp. 553–556, 1988.

    Google Scholar 

  10. A. Viterbi, “Error bounds of convolutional codes and an asymmetrically optimum decoding algorithm”, IEEE Transactions on Information Theory, IT-13, pp. 260–267, 1967.

    Google Scholar 

  11. A. Waibel, T. Hanazawa, G. Hinton, K. Shikano, and K. Lang, “Phoneme recognition using time-delay neural networks”, IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 37, no. 3, pp. 328–339, 1989.

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2002 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Lee, S., Yook, D. (2002). Audio-to-Visual Conversion Using Hidden Markov Models. In: Ishizuka, M., Sattar, A. (eds) PRICAI 2002: Trends in Artificial Intelligence. PRICAI 2002. Lecture Notes in Computer Science(), vol 2417. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45683-X_60

Download citation

  • DOI: https://doi.org/10.1007/3-540-45683-X_60

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-44038-3

  • Online ISBN: 978-3-540-45683-4

  • eBook Packages: Springer Book Archive