A Clustering Approach to Assess Real User Profiles in Spoken Dialogue Systems
Abstract
Evaluation methodologies for spoken dialogue systems try to provide an efficient means of assessing the quality of the system and/or predicting the user satisfaction. In order to do so, they must be carried out over a corpus of dialogues which contains as many possible prospective or real user types as possible. In this paper we present a clustering approach to provide insight on whether user profiles can be automatically detected from the interaction parameters and overall quality predictions, providing a way of corroborating the most representative features for defining user profiles. We have carried out different experiments over a corpus of 62 dialogues with the INSPIRE dialogue system, from which the clustering approach provided an efficient way of easily obtaining information about the suitability of distinguishing between different user groups to complete a more significative evaluation of the system.
Notes
Acknowledgements
Research funded by the Spanish project ASIES TIN2010-17344.
References
- 1.Callejas, Z., Griol, D., Engelbrecht, K.P.: Assessment of user simulators for spoken dialogue systems by means of subspace multidimensional clustering. In: Interspeech (2012)Google Scholar
- 2.Chandramohan, S., Geist, M., Lefevre, F., Pietquin, O.: Clustering Behaviors Of Spoken Dialogue Systems Users. In: ICASSP (2012)Google Scholar
- 3.Engelbrecht, K.P., Quade, M., Moeller, S.: Analysis of a new simulation approach to dialog system evaluation. Speech Comm. 51, 1234–1252 (2009)CrossRefGoogle Scholar
- 4.Espejo, G., Aztiria, A., Augusto, J.C., López-Cózar, R.: Creating adaptive intelligent environments by means of multimodal dialogue and learning systems. In: HCIAmI’11, pp. 362–373. Nottingham (2011)Google Scholar
- 5.Helal, A.: The engineering handbook of smart technology for aging, disability and independence. Wiley (2008)Google Scholar
- 6.Lucas-Cuesta, J., Ferreiros, J., Aztiria, A., Augusto, J., McTear, M.: Dialogue-based management of user feedback in an autonomous preference learning system. In: ICAART, pp. 330–336 (2010)Google Scholar
- 7.Moeller, S., Engelbrecht, K., Schleicher, R.: Predicting the quality and usability of spoken dialogue services. Speech Comm. 8–9 (2009)Google Scholar
- 8.Pelleg, D., Moore, A.W.: X-means: Extending K-means with Efficient Estimation of the Number of Clusters. In: ICML, pp. 727–734 (2000)Google Scholar
- 9.Raux, A., Langner, B., Black, A., Eskenazi, M.: Let’s go: Improving spoken dialog systems for the elderly and non-natives. In: Eurospeech (2003)Google Scholar
- 10.Rieser, V., Lemon, O.: Cluster-based User Simulations for Learning Dialogue Strategies. In: Interspeech, pp. 1766–1769 (2006)Google Scholar