Audio Visual Speaker Verification Based on Hybrid Fusion of Cross Modal Features

  • Girija Chetty
  • Michael Wagner
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4815)


In this paper, we propose hybrid fusion of audio and explicit correlation features for speaker identity verification applications. Experiments were performed with the GMM based speaker models with a hybrid fusion technique involving late fusion of explicit cross-modal fusion features, with implicit eigen lip and audio MFCC features. An evaluation of the system performance with different gender specific datasets from controlled VidTIMIT data base and opportunistic UCBN database shows a significant performance improvement.


Audio-visual speaker identity verification liveness checking cross modal correlations 


  1. 1.
    Brunelli, R., Falavigna, D.: Person Identification Using Multiple Cues. IEEE Transactions on Pattern Analysis and Machine Intelligence 17, 955–966 (1995)CrossRefGoogle Scholar
  2. 2.
    Kuratate, T., Munhall, K.G., Rubin, P.E., Vatikiotis-Bateson, E., Yehia, H.: Audio-visual synthesis of talking faces from speech production correlates. In: Proc. EuroSpeech 1999, ESCA (1999)Google Scholar
  3. 3.
    Maeda, S.: A face model derived from a guided PCA of motion capture data and McGurk effects. In: Proceedings of the ATR symposium on Cross-modal Processing of Faces and Voices, pp. 63–64 (January 2005)Google Scholar
  4. 4.
    Sanderson, C., Paliwal, K.K.: Fast features for face authentication under illumination direction changes. Pattern Recognition Letters 24, 2409–2419 (2003)CrossRefGoogle Scholar
  5. 5.
    Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis. Journal of the American Society for Information Science 41(6), 391–407 (1990)CrossRefGoogle Scholar
  6. 6.
    Borga, M., Knutsson, H.: Finding Efficient Nonlinear Visual Operators using Canonical Correlation Analysis. In: Proc. of SSAB 2000, Halmstad, pp. 13–16 (2000)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Girija Chetty
    • 1
  • Michael Wagner
    • 1
  1. 1.School of Information Sciences and Engineering, University of CanberraAustralia

Personalised recommendations