Abstract
The interaction quality paradigm has been suggested as evaluation method for spoken dialogue systems and several experiments based on the LEGO corpus have shown its suitability. However, the corpus size was rather limited resulting in insufficient data for some mathematical models. Hence, we present an extension to the LEGO corpus. We validate the annotation process and further show that applying support vector machine estimation results in similar performance on the original, the new and the combined data. Finally, we test previous statements about applying a Conditioned Hidden Markov Model or Rule Induction classification using the new data set.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
LEGOext and LEGO are publicly available under http://nt.uni-ulm.de/ds-lego.
- 2.
UAR, κ, and ρ are defined in Sect. 4.4.1
- 3.
UAR, κ, and ρ are defined in Sect. 4.4.1.
- 4.
UAR, κ, and ρ are defined in Sect. 4.4.1.
- 5.
Majority baseline means that the majority class is always predicted. This would result in a UAR of 0.2 for a five class problem.
References
Cohen J (1960) A coefficient of agreement for nominal scales. In: Educational and psychological measurement, vol 20, pp 37–46
Cohen J (1968) Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychol Bull 70(4):213
Cohen WW (1995) Fast effective rule induction. In: Proceedings of the 12th international conference on machine learning. Morgan Kaufmann, San Francisco, pp 115–123
El Asri L, Khouzaimi H, Laroche R, Pietquin O (2014) Ordinal regression for interaction quality prediction. In: IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, Florence, pp 3245–3249
Raux A, Bohus D, Langner B, Black AW, Eskenazi M (2006) Doing research on a deployed spoken dialogue system: one year of let’s go! experience. In: Proc. of the international conference on speech and language processing (ICSLP)
Schmitt A, Schatz B, Minker W (2011) Modeling and predicting quality in spoken human-computer interaction. In: Proceedings of the SIGDIAL 2011 conference. Association for Computational Linguistics, Portland, pp 173–184
Schmitt A, Ultes S, Minker W (2012) A parameterized and annotated spoken dialog corpus of the cmu let’s go bus information system. In: International conference on language resources and evaluation (LREC), pp 3369–337
Spearman CE (1904) The proof and measurement of association between two things. Am J Psychol 15:88–103
Ultes S, Minker W (2013a) Improving interaction quality recognition using error correction. In: Proceedings of the 14th annual meeting of the special interest group on discourse and dialogue. Association for Computational Linguistics, Metz, pp 122–126. http://www.aclweb.org/anthology/W/W13/W13-4018
Ultes S, Minker W (2013b) Interaction quality: a review. SibSAU (as in Siberian State Aerospace University) Newspaper 4:153–156. http://www.vestnik.sibsau.ru/images/vestnik/ves450.pdf
Ultes S, Minker W (2014) Interaction quality estimation in spoken dialogue systems using hybrid-hmms. In: Proceedings of the 15th annual meeting of the special interest group on discourse and dialogue (SIGDIAL). Association for Computational Linguistics, Philadelphia, pp 208–217. http://www.aclweb.org/anthology/W14-4328
Ultes S, Heinroth T, Schmitt A, Minker W (2011) A theoretical framework for a user-centered spoken dialog manager. In: Proceedings of the paralinguistic information and its integration in spoken dialogue systems workshop. Springer, New York, pp. 241–246
Ultes S, ElChabb R, Minker W (2012a) Application and evaluation of a conditioned hidden markov model for estimating interaction quality of spoken dialogue systems. In: Mariani J, Devillers L, Garnier-Rizet M, Rosset S (eds) Proceedings of the 4th international workshop on spoken language dialog system (IWSDS). Springer, New York, pp 141–150
Ultes S, Schmitt A, Minker W (2012b) Towards quality-adaptive spoken dialogue management. In: NAACL-HLT workshop on future directions and needs in the spoken dialog community: tools and data (SDCTD 2012). Association for Computational Linguistics, Montréal, pp 49–52. http://www.aclweb.org/anthology/W12-1819
Ultes S, ElChabb R, Schmitt A, Minker W (2013a) Jachmm: a java-based conditioned hidden markov model library. In: IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, Vancouver, pp 3213–3217
Ultes S, Schmitt A, Minker W (2013b) On quality ratings for spoken dialogue systems – experts vs. users. In: Proceedings of the 2013 conference of the North American chapter of the association for computational linguistics: human language technologies. Association for Computational Linguistics, Atlanta, pp 569–578
Ultes S, Dikme H, Minker W (2014a) Dialogue management for user-centered adaptive dialogue. In: Proceedings of the 5th international workshop on spoken dialogue systems (IWSDS)
Ultes S, Dikme H, Minker W (2014b) First insight into quality-adaptive dialogue. In: International conference on language resources and evaluation (LREC), pp 246–251
Vapnik VN (1995) The nature of statistical learning theory. Springer, New York
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Ultes, S., Platero Sánchez, M.J., Schmitt, A., Minker, W. (2015). Analysis of an Extended Interaction Quality Corpus. In: Lee, G., Kim, H., Jeong, M., Kim, JH. (eds) Natural Language Dialog Systems and Intelligent Assistants. Springer, Cham. https://doi.org/10.1007/978-3-319-19291-8_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-19291-8_4
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-19290-1
Online ISBN: 978-3-319-19291-8
eBook Packages: Computer ScienceComputer Science (R0)