Analysis of an Extended Interaction Quality Corpus

  • Stefan UltesEmail author
  • María Jesús Platero Sánchez
  • Alexander Schmitt
  • Wolfgang Minker


The interaction quality paradigm has been suggested as evaluation method for spoken dialogue systems and several experiments based on the LEGO corpus have shown its suitability. However, the corpus size was rather limited resulting in insufficient data for some mathematical models. Hence, we present an extension to the LEGO corpus. We validate the annotation process and further show that applying support vector machine estimation results in similar performance on the original, the new and the combined data. Finally, we test previous statements about applying a Conditioned Hidden Markov Model or Rule Induction classification using the new data set.


Automatic dialoge systems evaluation Statistical classification Support vector machine Hidden markov model 


  1. Cohen J (1960) A coefficient of agreement for nominal scales. In: Educational and psychological measurement, vol 20, pp 37–46CrossRefGoogle Scholar
  2. Cohen J (1968) Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychol Bull 70(4):213CrossRefGoogle Scholar
  3. Cohen WW (1995) Fast effective rule induction. In: Proceedings of the 12th international conference on machine learning. Morgan Kaufmann, San Francisco, pp 115–123Google Scholar
  4. El Asri L, Khouzaimi H, Laroche R, Pietquin O (2014) Ordinal regression for interaction quality prediction. In: IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, Florence, pp 3245–3249Google Scholar
  5. Raux A, Bohus D, Langner B, Black AW, Eskenazi M (2006) Doing research on a deployed spoken dialogue system: one year of let’s go! experience. In: Proc. of the international conference on speech and language processing (ICSLP)Google Scholar
  6. Schmitt A, Schatz B, Minker W (2011) Modeling and predicting quality in spoken human-computer interaction. In: Proceedings of the SIGDIAL 2011 conference. Association for Computational Linguistics, Portland, pp 173–184Google Scholar
  7. Schmitt A, Ultes S, Minker W (2012) A parameterized and annotated spoken dialog corpus of the cmu let’s go bus information system. In: International conference on language resources and evaluation (LREC), pp 3369–337Google Scholar
  8. Spearman CE (1904) The proof and measurement of association between two things. Am J Psychol 15:88–103Google Scholar
  9. Ultes S, Minker W (2013a) Improving interaction quality recognition using error correction. In: Proceedings of the 14th annual meeting of the special interest group on discourse and dialogue. Association for Computational Linguistics, Metz, pp 122–126.
  10. Ultes S, Minker W (2013b) Interaction quality: a review. SibSAU (as in Siberian State Aerospace University) Newspaper 4:153–156.
  11. Ultes S, Minker W (2014) Interaction quality estimation in spoken dialogue systems using hybrid-hmms. In: Proceedings of the 15th annual meeting of the special interest group on discourse and dialogue (SIGDIAL). Association for Computational Linguistics, Philadelphia, pp 208–217.
  12. Ultes S, Heinroth T, Schmitt A, Minker W (2011) A theoretical framework for a user-centered spoken dialog manager. In: Proceedings of the paralinguistic information and its integration in spoken dialogue systems workshop. Springer, New York, pp. 241–246Google Scholar
  13. Ultes S, ElChabb R, Minker W (2012a) Application and evaluation of a conditioned hidden markov model for estimating interaction quality of spoken dialogue systems. In: Mariani J, Devillers L, Garnier-Rizet M, Rosset S (eds) Proceedings of the 4th international workshop on spoken language dialog system (IWSDS). Springer, New York, pp 141–150Google Scholar
  14. Ultes S, Schmitt A, Minker W (2012b) Towards quality-adaptive spoken dialogue management. In: NAACL-HLT workshop on future directions and needs in the spoken dialog community: tools and data (SDCTD 2012). Association for Computational Linguistics, Montréal, pp 49–52.
  15. Ultes S, ElChabb R, Schmitt A, Minker W (2013a) Jachmm: a java-based conditioned hidden markov model library. In: IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, Vancouver, pp 3213–3217Google Scholar
  16. Ultes S, Schmitt A, Minker W (2013b) On quality ratings for spoken dialogue systems – experts vs. users. In: Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: human language technologies. Association for Computational Linguistics, Atlanta, pp 569–578Google Scholar
  17. Ultes S, Dikme H, Minker W (2014a) Dialogue management for user-centered adaptive dialogue. In: Proceedings of the 5th international workshop on spoken dialogue systems (IWSDS)Google Scholar
  18. Ultes S, Dikme H, Minker W (2014b) First insight into quality-adaptive dialogue. In: International conference on language resources and evaluation (LREC), pp 246–251Google Scholar
  19. Vapnik VN (1995) The nature of statistical learning theory. Springer, New YorkCrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Stefan Ultes
    • 1
    Email author
  • María Jesús Platero Sánchez
    • 2
  • Alexander Schmitt
    • 1
  • Wolfgang Minker
    • 1
  1. 1.Ulm UniversityUlmGermany
  2. 2.University of GranadaGranadaSpain

Personalised recommendations