Online Speaker Adaptation of an Acoustic Model Using Face Recognition

  • Pavel Campr
  • Aleš Pražák
  • Josef V. Psutka
  • Josef Psutka
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8082)


We have proposed and evaluated a novel approach for online speaker adaptation of an acoustic model based on face recognition. Instead of traditionally used audio-based speaker identification we investigated the video modality for the task of speaker detection. A simulated on-line transcription created by a Large-Vocabulary Continuous Speech Recognition (LVCSR) system for online subtitling is evaluated utilizing speaker independent acoustic models, gender dependent models and models of particular speakers. In the experiment, the speaker dependent acoustic models were trained offline, and are switched online based on the decision of a face recognizer, which reduced Word Error Rate (WER) by 12% relatively compared to speaker independent baseline system.


acoustic model speaker adaptation face recognition multimodal processing automatic speech recognition 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Pražák, A., Zajíc, Z., Machlica, L., Psutka, J.V.: Fast Speaker Adaptation in Automatic Online Subtitling. In: International Conference on Signal Processing and Multimedia Applications, vol.(1), pp. 126–130 (2009)Google Scholar
  2. 2.
    Ajmera, J., McCowan, I., Bourlard, H.: Robust Speaker Change Detection. IEEE Signal Processing Letters 11(8), 649–651 (2004)CrossRefGoogle Scholar
  3. 3.
    Pražák, A., Psutka, J.V., Hoidekr, J., Kanis, J., Müller, L., Psutka, J.: Automatic Online Subtitling of the Czech Parliament Meetings. In: Sojka, P., Kopeček, I., Pala, K. (eds.) TSD 2006. LNCS (LNAI), vol. 4188, pp. 501–508. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  4. 4.
    Trmal, J., Pražák, A., Loose, Z., Psutka, J.: Online TV Captioning of Czech Parliamentary Sessions. In: Sojka, P., Horák, A., Kopeček, I., Pala, K. (eds.) TSD 2010. LNCS (LNAI), vol. 6231, pp. 416–422. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  5. 5.
    Psutka, J.V., Vaněk, J., Psutka, J.: Speaker-clustered Acoustic Models Evaluated on GPU for on-line Subtitling of Parliament Meetings. In: Habernal, I., Matoušek, V. (eds.) TSD 2011. LNCS (LNAI), vol. 6836, pp. 284–290. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  6. 6.
    Povey, D.: Discriminative Training for Large Vocabulary Speech Recognition. PhD thesis, Cambridge University, Engineering Department (2003)Google Scholar
  7. 7.
    Zajíc, Z., Machlica, L., Müller, L.: Robust Statistic Estimates for Adaptation in the Task of Speech Recognition. In: Sojka, P., Horák, A., Kopeček, I., Pala, K. (eds.) TSD 2010. LNCS (LNAI), vol. 6231, pp. 464–471. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  8. 8.
    Uřičář, M., Franc, V., Hlaváč, V.: Detector of Facial Landmarks Learned by the Structured Output SVM. In: VISAPP 2012: Proceedings of the 7th International Conference on Computer Vision Theory and Applications1, Rome, Italy, pp. 547–556 (2012)Google Scholar
  9. 9.
    Sonnenburg, S., Franc, V.: COFFIN: A Computational Framework for Linear SVMs. Technical Report 1, Center for Machine Perception, Czech Technical University, Prague, Czech Republic (2009)Google Scholar
  10. 10.
    Franc, V., Sonneburg, S.: Optimized Cutting Plane Algorithm for Large-Scale Risk Minimization. Journal of Machine Learning Research 10, 2157–2192 (2009)zbMATHGoogle Scholar
  11. 11.
    El Khoury, E., Sénac, C., Joly, P.: Audiovisual diarization of people in video content. In: Multimedia Tools and Applications, pp. 1–29 (2012)Google Scholar
  12. 12.
    Bendris, M., Charlet, D., Chollet, G.: People indexing in TV-content using lip-activity and unsupervised audio-visual identity verification. In: 2011 9th International Workshop on Content-Based Multimedia Indexing (CBMI), pp. 139–144. IEEE (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Pavel Campr
    • 1
  • Aleš Pražák
    • 2
  • Josef V. Psutka
    • 2
  • Josef Psutka
    • 2
  1. 1.Center for Machine Perception, Department of Cybernetics, Faculty of Electrical EngineeringCzech Technical University in PraguePrague 6Czech Republic
  2. 2.Department of Cybernetics, Faculty of Applied SciencesUniversity of West Bohemia in PilsenPilsenCzech Republic

Personalised recommendations