Abstract
Evaluation methodologies for spoken dialogue systems try to provide an efficient means of assessing the quality of the system and/or predicting the user satisfaction. In order to do so, they must be carried out over a corpus of dialogues which contains as many possible prospective or real user types as possible. In this paper we present a clustering approach to provide insight on whether user profiles can be automatically detected from the interaction parameters and overall quality predictions, providing a way of corroborating the most representative features for defining user profiles. We have carried out different experiments over a corpus of 62 dialogues with the INSPIRE dialogue system, from which the clustering approach provided an efficient way of easily obtaining information about the suitability of distinguishing between different user groups to complete a more significative evaluation of the system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
For illustration purposes, we have grouped the five categories into three: bad&poor, fair, and good&excellent.
References
Callejas, Z., Griol, D., Engelbrecht, K.P.: Assessment of user simulators for spoken dialogue systems by means of subspace multidimensional clustering. In: Interspeech (2012)
Chandramohan, S., Geist, M., Lefevre, F., Pietquin, O.: Clustering Behaviors Of Spoken Dialogue Systems Users. In: ICASSP (2012)
Engelbrecht, K.P., Quade, M., Moeller, S.: Analysis of a new simulation approach to dialog system evaluation. Speech Comm. 51, 1234–1252 (2009)
Espejo, G., Aztiria, A., Augusto, J.C., López-Cózar, R.: Creating adaptive intelligent environments by means of multimodal dialogue and learning systems. In: HCIAmI’11, pp. 362–373. Nottingham (2011)
Helal, A.: The engineering handbook of smart technology for aging, disability and independence. Wiley (2008)
Lucas-Cuesta, J., Ferreiros, J., Aztiria, A., Augusto, J., McTear, M.: Dialogue-based management of user feedback in an autonomous preference learning system. In: ICAART, pp. 330–336 (2010)
Moeller, S., Engelbrecht, K., Schleicher, R.: Predicting the quality and usability of spoken dialogue services. Speech Comm. 8–9 (2009)
Pelleg, D., Moore, A.W.: X-means: Extending K-means with Efficient Estimation of the Number of Clusters. In: ICML, pp. 727–734 (2000)
Raux, A., Langner, B., Black, A., Eskenazi, M.: Let’s go: Improving spoken dialog systems for the elderly and non-natives. In: Eurospeech (2003)
Rieser, V., Lemon, O.: Cluster-based User Simulations for Learning Dialogue Strategies. In: Interspeech, pp. 1766–1769 (2006)
Acknowledgements
Research funded by the Spanish project ASIES TIN2010-17344.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer Science+Business Media New York
About this paper
Cite this paper
Callejas, Z., Griol, D., Engelbrecht, KP., López-Cózar, R. (2014). A Clustering Approach to Assess Real User Profiles in Spoken Dialogue Systems. In: Mariani, J., Rosset, S., Garnier-Rizet, M., Devillers, L. (eds) Natural Interaction with Robots, Knowbots and Smartphones. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-8280-2_29
Download citation
DOI: https://doi.org/10.1007/978-1-4614-8280-2_29
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-8279-6
Online ISBN: 978-1-4614-8280-2
eBook Packages: EngineeringEngineering (R0)