Assessing User Expertise in Spoken Dialog System Interactions

  • Eugénio Ribeiro
  • Fernando Batista
  • Isabel Trancoso
  • José Lopes
  • Ricardo Ribeiro
  • David Martins de Matos
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10077)


Identifying the level of expertise of its users is important for a system since it can lead to a better interaction through adaptation techniques. Furthermore, this information can be used in offline processes of root cause analysis. However, not much effort has been put into automatically identifying the level of expertise of an user, especially in dialog-based interactions. In this paper we present an approach based on a specific set of task related features. Based on the distribution of the features among the two classes – Novice and Expert – we used Random Forests as a classification approach. Furthermore, we used a Support Vector Machine classifier, in order to perform a result comparison. By applying these approaches on data from a real system, Let’s Go, we obtained preliminary results that we consider positive, given the difficulty of the task and the lack of competing approaches for comparison.


User expertise Let’s Go SVM Random Forest 



This work was supported by national funds through Fundação para a Ciência e a Tecnologia (FCT) with reference UID/CEC/50021/2013, by Universidade de Lisboa, and by the EC H2020 project RAGE under grant agreement No 644187.


  1. 1.
    SoX - Sound eXchange, version 14.4.1.
  2. 2.
    Breiman, L.: Random Forests. Mach. Learn. 45(1), 5–32 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Cortes, C., Vapnik, V.: Support-vector networks. In: Machine Learning, pp. 273–297 (1995)Google Scholar
  4. 4.
    Dreyfus, H.L., Dreyfus, S.E.: Mind Over Machine: The Power of Human Intuition and Expertise in the Era of the Computer. The Free Press, New York (1986)Google Scholar
  5. 5.
    Eyben, F., Weninger, F., Gross, F., Schuller, B.: Recent developments in openSMILE, the Munich open-source multimedia feature extractor. In: Proceedings of the 21st ACM International Conference on Multimedia, pp. 835–838 (2013)Google Scholar
  6. 6.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The WEKA data mining software: an update. SIGKDD Explor. Newsl. 11(1), 10–18 (2009)CrossRefGoogle Scholar
  7. 7.
    Hassel, L., Hagen, E.: Adaptation of an automotive dialogue system to users’ expertise and evaluation of the system. Lang. Resour. Eval. 40(1), 67–85 (2006)CrossRefGoogle Scholar
  8. 8.
    Hjalmarsson, A.: Adaptive Spoken Dialogue Systems (2005). Accessed 18 Dec 2015
  9. 9.
    Hjalmarsson, A.: Towards user modelling in conversational dialogue systems: a qualitative study of the dynamics of dialogue parameters. In: Proceedings of INTERSPEECH 2005, pp. 869–872 (2005)Google Scholar
  10. 10.
    Jokinen, K., Kanto, K.: User expertise modeling and adaptivity in a speech-based e-mail system. In: Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, pp. 87–94 (2004)Google Scholar
  11. 11.
    Lopes, J., Chorianopoulou, A., Palogiannidi, E., Moniz, H., Abad, A., Louka, K., Iosif, E., Potamianos, A.: The SpeDial datasets: datasets for spoken dialogue system analytics. In: Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC) (2016)Google Scholar
  12. 12.
    Meinedo, H., Viveiros, M., Neto, J.P.: Evaluation of a Live Broadcast News Subtitling System for Portuguese. In: Proceedings of INTERSPEECH 2008, pp. 508–511 (2008)Google Scholar
  13. 13.
    Nielsen, J.: Usability Engineering. Morgan Kaufmann Publishers Inc., San Francisco (1993)zbMATHGoogle Scholar
  14. 14.
    Platt, J.: Fast training of support vector machines using sequential minimal optimization. In: Advances in Kernel Methods - Support Vector Learning. MIT Press (1998)Google Scholar
  15. 15.
    Raux, A., Bohus, D., Langner, B., Black, A.W., Eskenazi, M.: Doing research on a deployed spoken dialogue system: one year of Lets Go! experience. In: Proceedings of INTERSPEECH 2006, pp. 65–68 (2006)Google Scholar
  16. 16.
    Schmitt, A., Ultes, S., Minker, W.: A parameterized and annotated spoken dialog corpus of the CMU Let’s Go bus information system. In: Proceedings of the 8th International Conference on Language Resources and Evaluation (LREC) (2012)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Eugénio Ribeiro
    • 1
    • 2
  • Fernando Batista
    • 1
    • 3
  • Isabel Trancoso
    • 1
    • 2
  • José Lopes
    • 4
  • Ricardo Ribeiro
    • 1
    • 3
  • David Martins de Matos
    • 1
    • 2
  1. 1.L2F – Spoken Language Systems LaboratoryINESC-ID LisboaLisbonPortugal
  2. 2.Instituto Superior TécnicoUniversidade de LisboaLisbonPortugal
  3. 3.ISCTE-IUL – Instituto Universitário de LisboaLisbonPortugal
  4. 4.KTH Speech, Music, and HearingStockholmSweden

Personalised recommendations