Automatic Speech Recognition Using Missing Data Techniques: Handling of Real-World Data

  • Jort F. Gemmeke
  • Maarten Van Segbroeck
  • Yujun Wang
  • Bert Cranen
  • Hugo Van hamme


In this chapter, we investigate the performance of a missing data recognizer on real-world speech from the SPEECON and SpeechDat-Car databases. In previous work we hypothesized that in real-world speech, which is corrupted not only by environmental noise, but also by speaker, reverberation and channel effects, the ‘reliable’ features do not match an acoustic model trained on clean speech. In a series of experiments, we investigate the validity of this hypothesis and explore to what extent performance can be improved by combining MDT with three conventional techniques, viz. multi-condition training, dereverberation and feature enhancement. Our results confirm our hypothesis and show that the mismatch can be reduced by multi-condition training of the acoustic models and feature enhancement, and that these effects combine to some degree. Our experiments with dereverberation reveal that reverberation can have a major impact on recognition performance, but that MDT with a suitable missing data mask is capable of compensating both the environmental noise as well as the reverberation at once.


Finite Impulse Response Automatic Speech Recognition Acoustic Model Clean Speech Word Error Rate 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Spraak: Speech processing, recognition and automatic annotation kit. Website (1996).
  2. 2.
    ETSI standard document: Speech processing, transmission and quality aspects (STQ); distributed speech recognition; advanced front-end feature extraction algorithm; ES 202 050 V1.1.5 (2007)Google Scholar
  3. 3.
    Astudillo, R.F., Kolossa, D., Mandelartz, P., Orglmeister, R.: An uncertainty propagation approach to robust ASR using the ETSI advanced front-end. IEEE Journal of Selected Topics in Signal Processing 4, 824 833 (2010)Google Scholar
  4. 4.
    C. Cortes, C., Vapnik, V.: Support-vector networks. Machine Learning pp. 273–297 (1995)Google Scholar
  5. 5.
    Chang, C., Lin, C.: LIBSVM: A library for support vector machines (2001)Google Scholar
  6. 6.
    Cooke, M., Green, P., Crawford, M.: Handling missing data in speech recognition. In: ICSLP-1994, pp. 1555–1558 (1994)Google Scholar
  7. 7.
    Cooke, M., Green, P., Josifovksi, L., Vizinho, A.: Robust automatic speech recognition with missing and unreliable acoustic data. Speech Communication 34, 267–285 (2001)MATHCrossRefGoogle Scholar
  8. 8.
    Delcroix, M., Nakatani, T., Watanabe, S.: Static and dynamic variance compensation for recognition of reverberant speech with dereverberation preprocessing. IEEE Transactions on Audio, Speech, and Language Processing 17(2), 324–334 (2009)CrossRefGoogle Scholar
  9. 9.
    Demange, S., Cerisara, C., Haton, J.P.: Accurate marginalization range for missing data recognition. In: Proc. of Interspeech, pp. 27–31 (2007)Google Scholar
  10. 10.
    Demuynck, K., Duchateau, J., Compernolle, D.V.: Optimal feature sub-space selection based on discriminant analysis. In: Proc. of European Conference on Speech Communication and Technology, vol. 3, pp. 1311–1314 (1999)Google Scholar
  11. 11.
    Duchateau, J., Demuynck, K., Compernolle, D.V.: Fast and accurate acoustic modelling with semicontinuous HMMs. Speech Communication 24(1), 5–17 (1998)CrossRefGoogle Scholar
  12. 12.
    Duchateau, J., Demuynck, K., Wambacq, D.V.C.P.: Improved parameter tying for efficient acoustic model evaluation in large vocabulary continuous speech recognition. In: Proc. ICSLP, vol. V, pp. 2215–2218. Sydney, Australia (1998)Google Scholar
  13. 13.
    Gemmeke, J.F., Wang, Y., Van Segbroeck, M., Cranen, B., Van hamme, H.: Application of noise robust MDT speech recognition on the SPEECON and SpeechDat-Car databases. In: Proc. of Interspeech. Brighton, UK (2009)Google Scholar
  14. 14.
    Hirsch, H., Pearce, D.: The aurora experimental framework for the performance evaluation of speech recognition systems under noisy conditions. In: Proc. of ISCA ASR2000 Workshop, pp. 181–188 (2000)Google Scholar
  15. 15.
    Huang, X., Acero, A., Hon, H.W.: Spoken Language Processing: A Guide to Theory, Algorithm and System Development. Prentice Hall (2001)Google Scholar
  16. 16.
    Iskra, D., Grosskopf, B., Marasek, K., van den Heuvel, H., Diehl, F., Kiessling, A.: SPEECON — speech databases for consumer devices: Database specification and validation. In: Proc. of LREC, pp. 329–333 (2002)Google Scholar
  17. 17.
    Josifovski, L., Cooke, M., Green, P., Vizinho, A.: State based imputation of missing data for robust speech recognitionand speech enhancement. In: Proc. of Eurospeech (1999)Google Scholar
  18. 18.
    Kamath, S., Loizou, P.: A multi-band spectral subtraction method for enhancing speech. In: Proc. of ICASSP (2002)Google Scholar
  19. 19.
    Kim, W., Hansen, J.H.L.: Time-frequency correlation-based missing-feature reconstruction for robust speech recognition in band-restricted conditions. IEEE Transactions on Audio, Speech, and Language Processing 17(7), 1292–1304 (2009)CrossRefGoogle Scholar
  20. 20.
    Martin, R.: Noise power spectral density estimation based on optimal smoothing and minimum statistics. IEEE Transactions on Speech and Audio Processing 9, 504–512 (2001)CrossRefGoogle Scholar
  21. 21.
    Palomäki, K.J., Brown, G.J., Barker, J.: Techniques for handling convolutional distortion with “missing data” automatic speech recognition. Speech Communication 43, 123–142 (2004)CrossRefGoogle Scholar
  22. 22.
    Parihar, N., Picone, J.: An analysis of the Aurora large vocabulary evaluation. In: Proc. of Eurospeech, pp. 337–340 (2003)Google Scholar
  23. 23.
    Raj, B., Seltzer, M., Stern, R.: Reconstruction of missing features for robust speech recognition. Speech Communication 43, 275–296 (2004)CrossRefGoogle Scholar
  24. 24.
    Raj, B., Singh, R., Stern, R.: Inference of missing spectrographic features for robust automatic speech recognition. In: Proc. of International Conference on Spoken Language Processing, pp. 1491–1494 (1998)Google Scholar
  25. 25.
    Raj, B., Stern, R.: Missing-feature approaches in speech recognition. Signal Processing Magazine 22(5), 101–116 (2005)CrossRefGoogle Scholar
  26. 26.
    Ramírez, J., Górriz, J., Segura, J., Puntonet, C., Rubio, A.: Speech/non-speech discrimination based on contextual information integrated bispectrum LRT. In: IEEE Signal Processing Letters (2006)Google Scholar
  27. 27.
    Remes, U., Palomäki, K.J., Kurimo, M.: Missing feature reconstruction and acoustic model adaptation combined for large vocabulary continuous speech recognition. In: Proc. of EUSIPCO (2008)Google Scholar
  28. 28.
    Seltzer, M., Raj, B., Stern, R.: A Bayesian classifier for spectrographic mask estimation for missing feature speech recognition. Speech Communication 43, 379–393 (2004)CrossRefGoogle Scholar
  29. 29.
    Stouten, V.: Robust automatic speech recognition in time-varying environments. Ph.D. thesis, K. U. Leuven (2006)Google Scholar
  30. 30.
    van den Heuvel, H., Boudy, J., Comeyne, R., Communications, M.N.: The SpeechDat-Car multilingual speech databases for in-car applications. In: Proc. of the European Conference on Speech Communication and Technology, pp. 2279–2282 (1999)Google Scholar
  31. 31.
    Van hamme, H.: Robust speech recognition using missing feature theory in the cepstral or LDA domain. In: Proc. of European Conference on Speech Communication and Technology, pp. 3089–3092 (2003)Google Scholar
  32. 32.
    Van hamme, H.: Prospect features and their application to missing data techniques for robust speech recognition. In: Proc. of Interspeech, pp. 101–104 (2004)Google Scholar
  33. 33.
    Van hamme, H.: Robust speech recognition using cepstral domain missing data techniques and noisy masks. In: Proc. of ICASSP, vol. 1, pp. 213–216 (2004)Google Scholar
  34. 34.
    Van hamme, H.: Handling time-derivative features in a missing data framework for robust automatic speech recognition. In: Proc. of ICASSP (2006)Google Scholar
  35. 35.
    Van Segbroeck, M.: Robust large vocabulary continuous speech recognition using missing data techniques. Ph.D. thesis, K. U. Leuven (2010)Google Scholar
  36. 36.
    Van Segbroeck, M., Van hamme, H.: Handling convolutional noise in missing data automatic speech recognition. In: Proc. of ICASSP, pp. 2562–2565 (2006)Google Scholar
  37. 37.
    Van Segbroeck, M., Van hamme, H.: Vector-quantization based mask estimation for missing data automatic speech recognition. In: Proc. of ICSLP, pp. 910–913 (2007)Google Scholar
  38. 38.
    van Waterschoot, T., Rombouts, G., Verhoeve, P., Moonen, M.: Double-talk-robust prediction error identification algorithms for acoustic echo cancellation. IEEE Transactions on Signal Processing 55(3), 846–858 (2007)CrossRefMathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  • Jort F. Gemmeke
    • 1
  • Maarten Van Segbroeck
    • 2
  • Yujun Wang
    • 2
  • Bert Cranen
    • 1
  • Hugo Van hamme
    • 2
  1. 1.Dept. of LinguisticsRadboud University NijmegenNijmegenThe Netherlands
  2. 2.ESAT DepartmentKatholieke Universiteit LeuvenLeuvenBelgium

Personalised recommendations