Language Resources and Evaluation

, Volume 41, Issue 2, pp 129–146 | Cite as

Validation of phonetic transcriptions in the context of automatic speech recognition

  • Christophe Van Bael
  • Henk van den Heuvel
  • Helmer Strik
Article

Abstract

Some of the speech databases and large spoken language corpora that have been collected during the last fifteen years have been (at least partly) annotated with a broad phonetic transcription. Such phonetic transcriptions are often validated in terms of their resemblance to a handcrafted reference transcription. However, there are at least two methodological issues questioning this validation method. First, no reference transcription can fully represent the phonetic truth. This calls into question the status of such a transcription as a single reference for the quality of other phonetic transcriptions. Second, phonetic transcriptions are often generated to serve various purposes, none of which are considered when the transcriptions are compared to a reference transcription that was not made with the same purpose in mind. Since phonetic transcriptions are often used for the development of automatic speech recognition (ASR) systems, and since the relationship between ASR performance and a transcription’s resemblance to a reference transcription does not seem to be straightforward, we verified whether phonetic transcriptions that are to be used for ASR development can be justifiably validated in terms of their similarity to a purpose-independent reference transcription. To this end, we validated canonical representations and manually verified broad phonetic transcriptions of read speech and spontaneous telephone dialogues in terms of their resemblance to a handcrafted reference transcription on the one hand, and in terms of their suitability for ASR development on the other hand. Whereas the manually verified phonetic transcriptions resembled the reference transcription much closer than the canonical representations, the use of both transcription types yielded similar recognition results. The difference between the outcomes of the two validation methods has two implications. First, ASR developers can save themselves the effort of collecting expensive reference transcriptions in order to validate phonetic transcriptions of speech databases or spoken language corpora. Second, phonetic transcriptions should preferably be validated in terms of the application they will serve because a higher resemblance to a purpose-independent reference transcription is no guarantee for a transcription to be better suited for ASR development.

Keywords

Broad phonetic transcriptions Validation Automatic speech recognition 

Abbreviations

ASR

Automatic speech recognition

CGN

Corpus Gesproken Nederlands—Spoken Dutch Corpus

MPT

Manual phonetic transcription

RT

Reference transcription

WER

Word error rate

Notes

Acknowledgements

The work of Christophe Van Bael was funded by the Speech Technology Foundation (Stichting Spraaktechnologie, Utrecht, The Netherlands). The authors would like to thank Louis Pols, various colleagues at the Department of Language and Speech (now CLST) and three anonymous reviewers for their comments on previous versions of this paper.

References

  1. Baayen, R. H., Piepenbrock, R., & Gulikers, L. (1995). The CELEX lexical database (release 2). Philadelphia, PA: Linguistic Data Consortium, University of Pennsylvania.Google Scholar
  2. Binnenpoorte, D., & Cucchiarini, C. (2003). Phonetic transcription of large speech corpora: How to boost efficiency without affecting quality. In Proceedings of the 15th international congress of phonetic sciences (ICPhS), Barcelona, Spain, pp. 2981–2984.Google Scholar
  3. Binnenpoorte, D., Goddijn, S. M. A., & Cucchiarini, C. (2003). How to improve human and machine transcriptions of spontaneous speech. In Proceedings of the ISCA/IEEE workshop on spontaneous speech processing and recognition (SSPR), Tokyo, Japan, pp. 147–150.Google Scholar
  4. Booij, G. (1999). The phonology of Dutch. New York: Oxford University Press.Google Scholar
  5. CELEX Lexical database (2005). [http://www.ru.nl/celex/].Google Scholar
  6. Cucchiarini, C. (1993). Phonetic transcription: A methodological and empirical study. Ph.D. Dissertation, University of Nijmegen, the Netherlands.Google Scholar
  7. Cucchiarini, C. (1996). Assessing transcription agreement: Methodological aspects. Clinical Linguistics and Phonetics, 10(2), 131–155.CrossRefGoogle Scholar
  8. Goddijn, S. M. A., & Binnenpoorte, D. (2003). Assessing manually corrected broad phonetic transcriptions in the spoken Dutch Corpus. In Proceedings of the 15th international congress of phonetic sciences (ICPhS), Barcelona, Spain, pp. 1361–1364.Google Scholar
  9. Godfrey, J., Holliman, E., & McDaniel, J. (1992). SWITCHBOARD: Telephone speech corpus for research and development. In Proceedings of the IEEE international conference on acoustics, speech and signal processing (ICASSP), San Francisco, USA, pp. 737–740.Google Scholar
  10. Greenberg, S. (1997). The Switchboard Transcription Project. Research Report #24, 1996. Large vocabulary continuous speech recognition summer research workshop technical report series. Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD, USA.Google Scholar
  11. Hoste, V., Daelemans, W., Tjong Kim Sang, E., & Gillis, S. (2000). Meta-learning for phonemic annotation of corpora. In Proceedings of the 17th international conference on machine learning (ICML), Stanford University, CA, USA, pp. 375–382.Google Scholar
  12. Kerkhoff, J., & Rietveld, T. (1994). Prosody in Niros with Fonpars and Alfeios. In Proceedings of the Department of Language and Speech, Vol. 18, University of Nijmegen, pp. 107–119.Google Scholar
  13. Kessens, J. M., & Strik, H. (2004). On automatic phonetic transcription quality: Lower word error rates do not guarantee better transcriptions. Computer, Speech and Language, 18, 123–141.CrossRefGoogle Scholar
  14. Kipp, A., Wesenick, M.-B., & Schiel, F. (1996). Automatic detection and segmentation of pronunciation variants in German speech corpora. In Proceedings of the international conference on spoken language processing (ICSLP), Philadelphia, USA, pp. 106–109.Google Scholar
  15. Kuijpers, C., & van Donselaar, W. (1997). The influence of rhythmic context on schwa epenthesis and schwa deletion in Dutch. Language and Speech, 41(1), 87–108.Google Scholar
  16. Oostdijk, N. (2002). The design of the spoken Dutch corpus. In P. Peters, P. Collins, & A. Smith (Eds.), New frontiers of corpus research (pp. 105–112). Amsterdam: Rodopi.Google Scholar
  17. PAROLE lexicon. (2005). [http://ww2.tst.inl.nl].Google Scholar
  18. Pearce, D. (2001). Developing the ETSI Aurora advanced distributed speech recognition front-end & what next? In Proceedings of the IEEE workshop on automatic speech recognition and understanding (ASRU), Madonna di Campiglio Trento, Italy, pp. 131–134.Google Scholar
  19. Quazza, S., & van den Heuvel, H. (2000). Lexicon development for speech and language processing. In F. Van Eynde & D. Gibbon (Eds.), Lexicon development for speech and language processing (pp. 207–233). Dordrecht: Kluwer Academic Publishers.Google Scholar
  20. Referentiebestand Nederlands (RBN). (2005). [http://ww2.tst.inl.nl].Google Scholar
  21. Saraçlar, M., Nock, H., & Khudanpur, S. (2000). Pronunciation modeling by sharing Gaussian densities across phonetic models. Computer Speech and Language, 14, 137–160.CrossRefGoogle Scholar
  22. Shriberg, L. D., Kwiatkowski, J., & Hoffman, K. (1984). A procedure for phonetic transcription by consensus. Journal of Speech and Hearing Research, 27, 456–465.Google Scholar
  23. Spoken Dutch Corpus – Het Project Corpus Gesproken Nederlands. (2005). [http://lands.let.kun.nl/cgn/ehome.htm].Google Scholar
  24. Young, S., Evermann, G., Kershaw, D., Moore, G., Odell, J., Ollason, D., Valtchev, V., & Woodland, P. (2001). The HTK book (for HTK version 3.1). Cambridge University Engineering Department.Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2007

Authors and Affiliations

  • Christophe Van Bael
    • 1
  • Henk van den Heuvel
    • 1
  • Helmer Strik
    • 1
  1. 1.Centre for Language and Speech Technology Radboud University NijmegenNijmegenThe Netherlands

Personalised recommendations