Assessing User Expertise in Spoken Dialog System Interactions
Identifying the level of expertise of its users is important for a system since it can lead to a better interaction through adaptation techniques. Furthermore, this information can be used in offline processes of root cause analysis. However, not much effort has been put into automatically identifying the level of expertise of an user, especially in dialog-based interactions. In this paper we present an approach based on a specific set of task related features. Based on the distribution of the features among the two classes – Novice and Expert – we used Random Forests as a classification approach. Furthermore, we used a Support Vector Machine classifier, in order to perform a result comparison. By applying these approaches on data from a real system, Let’s Go, we obtained preliminary results that we consider positive, given the difficulty of the task and the lack of competing approaches for comparison.
KeywordsUser expertise Let’s Go SVM Random Forest
This work was supported by national funds through Fundação para a Ciência e a Tecnologia (FCT) with reference UID/CEC/50021/2013, by Universidade de Lisboa, and by the EC H2020 project RAGE under grant agreement No 644187.
- 1.SoX - Sound eXchange, version 14.4.1. http://sox.sourceforge.net/
- 3.Cortes, C., Vapnik, V.: Support-vector networks. In: Machine Learning, pp. 273–297 (1995)Google Scholar
- 4.Dreyfus, H.L., Dreyfus, S.E.: Mind Over Machine: The Power of Human Intuition and Expertise in the Era of the Computer. The Free Press, New York (1986)Google Scholar
- 5.Eyben, F., Weninger, F., Gross, F., Schuller, B.: Recent developments in openSMILE, the Munich open-source multimedia feature extractor. In: Proceedings of the 21st ACM International Conference on Multimedia, pp. 835–838 (2013)Google Scholar
- 8.Hjalmarsson, A.: Adaptive Spoken Dialogue Systems (2005). http://www.speech.kth.se/~rolf/NGSLT/gslt_papers_2004/annah_termpaper_05.pdf. Accessed 18 Dec 2015
- 9.Hjalmarsson, A.: Towards user modelling in conversational dialogue systems: a qualitative study of the dynamics of dialogue parameters. In: Proceedings of INTERSPEECH 2005, pp. 869–872 (2005)Google Scholar
- 10.Jokinen, K., Kanto, K.: User expertise modeling and adaptivity in a speech-based e-mail system. In: Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, pp. 87–94 (2004)Google Scholar
- 11.Lopes, J., Chorianopoulou, A., Palogiannidi, E., Moniz, H., Abad, A., Louka, K., Iosif, E., Potamianos, A.: The SpeDial datasets: datasets for spoken dialogue system analytics. In: Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC) (2016)Google Scholar
- 12.Meinedo, H., Viveiros, M., Neto, J.P.: Evaluation of a Live Broadcast News Subtitling System for Portuguese. In: Proceedings of INTERSPEECH 2008, pp. 508–511 (2008)Google Scholar
- 14.Platt, J.: Fast training of support vector machines using sequential minimal optimization. In: Advances in Kernel Methods - Support Vector Learning. MIT Press (1998)Google Scholar
- 15.Raux, A., Bohus, D., Langner, B., Black, A.W., Eskenazi, M.: Doing research on a deployed spoken dialogue system: one year of Lets Go! experience. In: Proceedings of INTERSPEECH 2006, pp. 65–68 (2006)Google Scholar
- 16.Schmitt, A., Ultes, S., Minker, W.: A parameterized and annotated spoken dialog corpus of the CMU Let’s Go bus information system. In: Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC) (2012)Google Scholar