Evaluation of ASR Systems, Algorithms and Databases

  • Gérard Chollet
Conference paper
Part of the NATO ASI Series book series (volume 147)


The evaluation of ASR technology and applications is approached from different perspectives. The recognition performance gives only a partial indication of user satisfaction: the entire manmachine interface should be considered. Many factors contribute to the variability of speech and affect the performance of recognizers. Some recommendations are expressed concerning the size and the content of databases distributed for training and test purposes. Techniques to model the variability of speech are proposed. They are implemented on a test workstation. Predicting the performance of a given recognizer in a particular situation is possible. It is argued that most of these techniques could also be adapted to improve the robustness of recognizers and speaker verifiers.


Speech Recognition Automatic Speech Recognition Speech Recognition System Speaker Verification Speech Recognizer 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    G. Chollet, A. Astier, and M. Rossi. Evaluating the performance of speech recognizers at the acoustic-phonetic level. In IEEE-IASSP, Denver, 1981.Google Scholar
  2. [2]
    G. Chollet, F. Çapman, Ph. Bardaud, and Ch. Tadj. Measurement of the limits of the reference recogniser SAMREC1: noise addition and simulation of the lombard effect. In Final report SAM, June 1992.Google Scholar
  3. [3]
    G. Chollet, F. Çapman, Ch. Mokbel, and J. Farah Abou Daoud. On the evaluation of recognizers: statistical validity of the tests. In Final report SAM, June 1992.Google Scholar
  4. [4]
    G. Chollet and C. Gagnoulet. Evaluating speech recognizers and data bases using a reference system. In IEEE-ICASSP, Paris, 1982.Google Scholar
  5. [5]
    L. Gillick and S J. Cox. Some statistical issues in the comparison of speech recognition algorithms. In IEEE-ICASSP, pages 532–535, Glasgow, 1989.Google Scholar
  6. [6]
    J.Ph. Goldman and G. Chollet. Voice tranformations for the evaluation of speaker verification systems. In NATO-ASI, Bubion, 1993.Google Scholar
  7. [7]
    W.A. Lea. Procedures for evaluating the performances of commercial speech recognizers. In Speech Science Publication, Santa Barbara, 1981.Google Scholar
  8. [8]
    W.A. Lea. What causes speech recognizers to make mistakes? IEEE-ICASSP, 3:2030–2033, 1982.Google Scholar
  9. [9]
    J.N. Marcus. Significance tests for comparing speech recognizer performance using small test sets. EUROSPEECH, 2:465–468, 1989.Google Scholar
  10. [10]
    C. Montacié and G. Chollet. Système de reference pour revaluation d’ applications et la caractérisation de bases de données en reconnaissance automatique de la parole. In l6ème JEP, Hammamet, 1987.Google Scholar
  11. [11]
    R. Moore. Evaluating speech recognizers. IEEE-ICASSP, pages 178–185, 1977.Google Scholar
  12. [12]
    D.S. Pallett. Performance assessment of automatic speech recognizers. Journal of Research of the NBS, 90:5:371–387, 1985.Google Scholar
  13. [13]
    SAM. Multi-lingual Speech input/output Assessment Methodology and Standardisation. Final report, June 1992.Google Scholar
  14. [14]
    H. Steeneken and J. Van Velden. RAMOS: Recogniser assessment by means of manipulation of speech. Proc. EUROSPEECH, pages 316–319, 1989.Google Scholar
  15. [15]
    M.M. Taylor. Issues in the evaluation of speech recognition systems. In NATO RSG-10, DCIEM, Downsview, 1980.Google Scholar
  16. [16]
    H. Valbret, E. Moulines, and J. P. Tubach. Voice transformation using psola technique. Speech Communication, 11:175–187, 1992.CrossRefGoogle Scholar
  17. [17]
    J. Woodard and J. Nelson. An information theoretic measure of speech recognition performance. Proc. Workshop on Standardizationfor Speech I/O, Mar. 1982.Google Scholar
  18. [18]
    S. J. Young. HTK: Hidden Markov Toolkit (Version 1.4). Cambridge University Engineering Department, Cambridge, England, 1991.Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 1995

Authors and Affiliations

  • Gérard Chollet
    • 1
    • 2
  1. 1.TELECOM-ParisParis, cedex 13France
  2. 2.DIAPParisFrance

Personalised recommendations