Advertisement

A Bayesian Approach to Audio-Visual Speaker Identification

  • Ara V. Nefian
  • Lu Hong Liang
  • Tieyan Fu
  • Xiao Xing Liu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2688)

Abstract

In this paper we describe a text dependent audio-visual speaker identification approach that combines face recognition and audio-visual speech-based identification systems. The temporal sequence of audio and visual observations obtained from the acoustic speech and the shape of the mouth are modeled using a set of coupled hidden Markov models (CHMM), one for each phoneme-viseme pair and for each person in the database. The use of CHMM in our system is justified by the capability of this model to describe the natural audio and visual state asynchrony as well as their conditional dependence over time. Next, the likelihood obtained for each person in the database is combined with the face recognition likelihood obtained using an embedded hidden Markov model (EHMM). Experimental results on XM2VTS database show that our system improves the accuracy of the audio-only or video-only speaker identification at all levels of acoustic signal-to-noise ratio (SNR) from 5 to 30db.

Keywords

Hide Markov Model Face Recognition Visual Speech Backbone Node Continuous Speech Recognition 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. [1]
    P.N. Belhumeur, J.P Hespanha, and D.J. Kriegman. Eigenfaces vs Fisherfaces: Recognition using class specific linear projection. In Proceedings of Fourth Europeean Conference on Computer Vision, ECCV’96, pages 45–58, April 1996.Google Scholar
  2. [2]
    M. Brand, N. Oliver, and A. Pentland. Coupled hidden Markov models for complex action recognition. In IEEE International Conference on Computer Vision and Pattern Recognition, pages 994–999, 1997.Google Scholar
  3. [3]
    C. Chibelushi, F. Deravi, and S.D. Mason. A review of speech-based bimodal recognition. IEEE Transactions on Multimedia, 4(1):23–37, March 2002.CrossRefGoogle Scholar
  4. [4]
    S. Dupont and J. Luettin. Audio-visual speech modeling for continuous speech recognition. IEEE Transactions on Multimedia, 2:141–151, September 2000.CrossRefGoogle Scholar
  5. [5]
    S. Kuo and O.E. Agazzi. Keyword spotting in poorly printed documents using pseudo 2-D Hidden Markov Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 16(8):842–848, August 1994.CrossRefGoogle Scholar
  6. [6]
    A. Lawrence, C.L. Giles, A.C Tsoi, and A.D. Back. Face recognition: A convolutional neural network approach. IEEE Transactions on Neural Networks, 8(1):98–113, 1997.CrossRefGoogle Scholar
  7. [7]
    Jia Lia, A. Najmi, and R.M. Gray. Image classification by a two-dimensional hidden markov model. IEEE Transactions on Signal Processing, 48(2):517–533, February 2000.CrossRefGoogle Scholar
  8. [8]
    L. Liang, X. Liu, X. Pi, Y. Zhao, and A. V. Nefian. Speaker independent audiovisual continuous speech recognition. In International Conference on Multimedia and Expo, volume 2, pages 25–28, 2002.Google Scholar
  9. [9]
    R. Lienhart and J. Maydt. An extened set of Haar-like features for rapid objection detection. In IEEE International Conference on Image Processing, volume 1, pages 900–903, 2002.Google Scholar
  10. [10]
    X. Liu, L. Liang, Y. Zhao, X. Pi, and A. V. Nefian. Audio-visual continuous speech recognition using a coupled hidden Markov model. In International Conference on Spoken Language Processing, 2002.Google Scholar
  11. [11]
    J. Luettin and G. Maitre. Evaluation protocol for the XM2FDB database. In IDIAP-COM 98-05, 1998.Google Scholar
  12. [12]
    A. V. Nefian and M. H. Hayes. Face recognition using an embedded HMM. In Proceedings of the IEEE Conference on Audio and Video-based Biometric Person Authentication, pages 19–24, March 1999.Google Scholar
  13. [13]
    A. V. Nefian, L. Liang, X. Pi, X. Liu, and C. Mao. A coupled hiddenMarkov model for audio-visual speech recognition. In International Conference on Acoustics, Speech and Signal Processing, volume 2, pages 2013–2016, 2002.Google Scholar
  14. [14]
    Ara V. Nefian. Embedded Bayesian networks for face recognition. In IEEE International Conference on Multimedia and Expo, volume 2, pages 25–28, 2002.Google Scholar
  15. [15]
    Ara V. Nefian and Monson H. Hayes III. Maximum likelihood training of the embedded HMM for face detection and recognition. In IEEE International Conference on Image Processing, volume 1, pages 33–36, 2000.Google Scholar
  16. [16]
    C. Neti, G. Potamianos, J. Luettin, I. Matthews, D. Vergyri, J. Sison, A. Mashari, and J. Zhou. Audio visual speech recognition. In Final Workshop 2000 Report, 2000.Google Scholar
  17. [17]
    Jonathon Phillips. Matching pursuit filters applied to face identification. IEEE Transactions on Image Processing, 7(8):1150–1164, August 1998.CrossRefGoogle Scholar
  18. [18]
    G. Potamianos, J. Luettin, and C. Neti. Asynchronous stream modeling for large vocabulary audio-visual speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing, volume 1, pages 169–172, 2001.Google Scholar
  19. [19]
    D.A. Reynolds, T.F. Quatieri, and R.B. Dunn. Speaker verification using an adapted gaussian mixture model. Digital Signal Processing, 10:19–41, 2000.CrossRefGoogle Scholar
  20. [20]
    F. Samaria. Face Recognition Using Hidden Markov Models. PhD thesis, University of Cambridge, 1994.Google Scholar
  21. [21]
    L.K. Saul and M.L. Jordan. Boltzmann chains and hidden Markov models. In G. Tesauro, David S. Touretzky, and T. Leen, editors, Advances in Neural Information Processing Systems, volume 7. The MIT Press, 1995.Google Scholar
  22. [22]
    S.B. Yacouband, S. Luettin, J. Jonsson, K. Matas, and J. Kittler. Audio-visual person verification. In IEEE International Conference on Computer Vision and Pattern Recognition, pages 580–585, 1999.Google Scholar
  23. [23]
    M. Turk and A.P. Pentland. Face recognition using eigenfaces. In Proceedings of International Conference on Pattern Recognition, pages 586–591, 1991.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Ara V. Nefian
    • 1
  • Lu Hong Liang
    • 1
  • Tieyan Fu
    • 2
  • Xiao Xing Liu
    • 1
  1. 1.Microprocessor Research LabsIntel CorporationUSA
  2. 2.Computer Science and Technology DepartmentTsinghua UniversityUSA

Personalised recommendations