Robust Methodology for TTS Enhancement Evaluation

  • Daniel Tihelka
  • Martin Grůber
  • Zdeněk Hanzlíček
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8082)


The paper points to problematic and usually neglected aspects of using listening tests for TTS evaluation. It shows that simple random selection of phrases to be listened to may not cover those cases which are relevant to the evaluated TTS system. Also, it shows that a reliable phrase set cannot be chosen without a deeper knowledge of the distribution of differences in synthetic speech, which are obtained by comparing the output generated by an evaluated TTS system to what stands as a baseline system. Having such knowledge, the method able to evaluate the reliability of listening tests, as related to the estimation of possible invalidity of listening results-derived conclusion, is proposed here and demonstrated on real examples.


speech synthesis TTS evaluation listening tests statistical reliability 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ircing, P., Psutka, J., Psutka, J.V.: Using morphological information for robust language modeling in Czech ASR system. IEEE Transactions on Audio Speech and Language Processing 17, 840–847 (2009)CrossRefGoogle Scholar
  2. 2.
    Pražák, A., Psutka, J.V., Hoidekr, J., Kanis, J., Müller, L., Psutka, J.: Automatic online subtitling of the Czech parliament meetings. In: Sojka, P., Kopeček, I., Pala, K. (eds.) TSD 2006. LNCS (LNAI), vol. 4188, pp. 501–508. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  3. 3.
    Hanzlíček, Z., Matoušek, J.: Voice conversion based on probabilistic parameter transformation and extended inter-speaker residual prediction. In: Matoušek, V., Mautner, P. (eds.) TSD 2007. LNCS (LNAI), vol. 4629, pp. 480–487. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  4. 4.
    Tihelka, D., Romportl, J.: Exploring automatic similarity measures for unit selection tuning. In: INTERSPEECH 2009, Proceedings of the 10th Annual Conference of International Speech Communication Association, pp. 736–739. ISCA, Brighton (2009)Google Scholar
  5. 5.
    Black, A.W., Tokuda, K.: The Blizzard Challenge – 2005: Evaluating corpus-based speech synthesis on common datasets. In: Interspeech, Lisbon, Portugal, pp. 77–80 (2005)Google Scholar
  6. 6.
    Yamagishi, J., Zen, H., Jian Wu, Y., Toda, T., Tokuda, K.: The HTS-2008 system: Yet another evaluation of the speaker-adaptive hmm-based speech synthesis system in The 2008 Blizzard Challenge. In: Proc. Blizzard Challenge 2008 (2008)Google Scholar
  7. 7.
    Tihelka, D., Kala, J., Matoušek, J.: Enhancements of viterbi search for fast unit selection synthesis. In: Interspeech, pp. 174–177 (2010)Google Scholar
  8. 8.
    Eide, E., Aaron, A., Bakis, R., Cohen, P., Donovan, R., Hamza, W., Mathes, T., Picheny, M., Polkosky, M., Smith, M., Viswanathan, M.: Recent improvements to the IBM trainable speech synthesis system. In: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2003, vol. 1, pp. 708–711 (2003)Google Scholar
  9. 9.
    Vepa, J., King, S.: Subjective evaluation of join cost functions used in unit selection speech synthesis. In: Interspeech, Jeju Island, Korea, pp. 1181–1184 (2004)Google Scholar
  10. 10.
    Grůber, M., Tihelka, D.: Expressive speech synthesis for Czech limited domain dialogue system – basic experiments. In: IEEE 10th International Conference on Signal Processing (ICSP), Beijing, China, vol. 1, pp. 561–564 (2010)Google Scholar
  11. 11.
    Grůber, M.: Enumerating differences between various communicative functions for purposes of Czech expressive speech synthesis in limited domain. In: Interspeech, pp. 650–653. Curran Associates, Inc., Morehouse Lane (2012)Google Scholar
  12. 12.
    Přibil, J., Přibilová, A.: Distributed listening test program for synthetic speech evaluation, pp. 241–242 (2008)Google Scholar
  13. 13.
    Hanzlíček, Z., Matoušek, J., Tihelka, D.: Experiments on reducing footprint of unit selection TTS system (2013)Google Scholar
  14. 14.
    Bowman, A.W., Azzalini, A.: Applied Smoothing Techniques for Data Analysis. Oxford University Press (1997)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Daniel Tihelka
    • 1
  • Martin Grůber
    • 1
  • Zdeněk Hanzlíček
    • 1
  1. 1.Faculty of Applied Sciences, Dept. of CyberneticsUniversity of West BohemiaPlzeňCzech Republic

Personalised recommendations