Recurrent Neural Network Interaction Quality Estimation

  • Louisa PragstEmail author
  • Stefan Ultes
  • Wolfgang Minker
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 427)


Getting a good estimation of the Interaction Quality (IQ) of a spoken dialogue helps to increase the user satisfaction as the dialogue strategy may be adapted accordingly. Therefore, some research has already been conducted in order to automatically estimate the Interaction Quality. This article adds to this by describing how Recurrent Neural Networks may be used to estimate the Interaction Quality for each dialogue turn and by evaluating their performance on this task. Here, we will show that RNNs may outperform non-recurrent neural networks.


RNN Sequential data Quality of dialogue Recurrent neural network Neural network Interaction quality User satisfaction Spoken dialogue system 



This paper is part of a project that has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 645012.


  1. 1.
    Ultes, S., Heinroth, T., Schmitt, A., Minker, W.: A theoretical framework for a user-centered spoken dialog manager. In: Proceedings of the Paralinguistic Information and its Integration in Spoken Dialogue Systems Workshop. pp. 241—246. Springer (2011)Google Scholar
  2. 2.
    Ultes, S., Dikme, H., Minker, W.: Dialogue management for user-centered adaptive dialogue. In: Proceedings of IWSDS (2014).
  3. 3.
    Ultes, S., Dikme, H., Minker, W.: First insight into quality-adaptive dialogue. In: Proceedings of the LREC, pp. 246–251 (2014)Google Scholar
  4. 4.
    Ultes, S., Kraus, M., Schmitt, A., Minker, W.: Quality-adaptive spoken dialogue initiative selection and implications on reward modelling. In: Proceedings of SIGDIAL, pp. 374–383. ACL (2015)Google Scholar
  5. 5.
    Ultes, S., Minker, W.: Improving interaction quality recognition using error correction. In: Proceedings of SIGDIAL, pp. 122–126. ACL (2013).
  6. 6.
    Ultes, S., Minker, W.: Interaction quality: a review. Bulletin of Siberian State Aerospace University named after academician M.F. Reshetnev (4), 153–156 (2013).
  7. 7.
    Ultes, S., Platero Sánchez, M.J., Schmitt, A., Minker, W.: Analysis of an extended interaction quality corpus. In: Proceedings of IWSDS (2015)Google Scholar
  8. 8.
    Ultes, S., ElChabb, R., Minker, W.: Application and evaluation of a conditioned hidden markov model for estimating interaction quality of spoken dialogue systems. In: Proceedings of IWSDS, pp. 141–150. Springer (2012)Google Scholar
  9. 9.
    Ultes, S., Minker, W.: Interaction quality estimation in spoken dialogue systems using hybrid-hmms. In: Proceedings of SIGDIAL, pp. 208–217. ACL (2014).
  10. 10.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  11. 11.
    Mesnil, G., He, X., Deng, L., Bengio, Y.: Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding. In: Proceedings of the INTERSPEECH, pp. 3771–3775 (2013)Google Scholar
  12. 12.
    Henderson, M., Thomson, B., Young, S.: Robust dialog state tracking using delexicalised recurrent neural networks and unsupervised adaptation. In: Proceedings of the SLT, pp. 360–365. IEEE (2014)Google Scholar
  13. 13.
    Wen, T.H., Gačić, M., Kim, D., Mrkšic, N., Su, P.H., Vandyke, D., Young, S.: Stochastic language generation in dialogue using recurrent neural networks with convolutional sentence reranking. In: Proceedings of the ACL, pp. 275–284. (2015)Google Scholar
  14. 14.
    Su, P.H., Vandyke, D., Gasic, M., Kim, D., Mrksic, N., Wen, T.H., Young, S.: Learning from real users: rating dialogue success with neural networks for reinforcement learning in spoken dialogue systems. arXiv:1508.03386 (2015)
  15. 15.
    Walker, M., Litman, D.J., Kamm, C.A., Abella, A.: Paradise: a framework for evaluating spoken dialogue agents. In: Proceedings of ACL, pp. 271–280. ACL, Morristown, NJ, USA (1997)Google Scholar
  16. 16.
    Engelbrecht, K.P., Gödde, F., Hartard, F., Ketabdar, H., Möller, S.: Modeling user satisfaction with hidden markov model. In: Proceedings of SIGDIAL, pp. 170–177. ACL, Morristown, NJ, USA (2009)Google Scholar
  17. 17.
    Higashinaka, R., Minami, Y., Dohsaka, K., Meguro, T.: Issues in predicting user satisfaction transitions in dialogues: Individual differences, evaluation criteria, and prediction models. In: Spoken Dialogue Systems for Ambient Environments, Lecture Notes in Computer Science, vol. 6392, pp. 48–60. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-16202-2_5
  18. 18.
    Hara, S., Kitaoka, N., Takeda, K.: Estimation method of user satisfaction using n-gram-based dialog history model for spoken dialog system. In: Proceedings of LREC. ELRA, Valletta, Malta (2010)Google Scholar
  19. 19.
    Schmitt, A., Ultes, S.: Interaction quality: assessing the quality of ongoing spoken dialog interaction by experts—and how it relates to user satisfaction. Speech Commun. (2015) accepted for publicationGoogle Scholar
  20. 20.
    Ultes, S., Schmitt, A., Minker, W.: Towards quality-adaptive spoken dialogue management. In: Proceedings of the NAACL-HLT: SDCTD, pp. 49–52. ACL, Montréal, Canada (2012).
  21. 21.
    Ultes, S., Schmitt, A., Minker, W.: On quality ratings for spoken dialogue systems—experts vs. users. In: Proceedings of NAACL-HLT, pp. 569–578. ACL (2013)Google Scholar
  22. 22.
    Schmitt, A., Schatz, B., Minker, W.: Modeling and predicting quality in spoken human-computer interaction. In: Proceedings of SIGDIAL, pp. 173–184. ACL, Portland, Oregon, USA (2011)Google Scholar
  23. 23.
    Raux, A., Bohus, D., Langner, B., Black, A.W., Eskenazi, M.: Doing research on a deployed spoken dialogue system: one year of let’s go! experience. In: Proceedings of ICSLP (2006)Google Scholar
  24. 24.
    Schmitt, A., Ultes, S., Minker, W.: A parameterized and annotated spoken dialog corpus of the CMU Let’s go bus information system. In: Proceedings of the LREC, pp. 3369–337 (2012)Google Scholar
  25. 25.
    Elman, J.L.: Finding structure in time. Cogn. Sci. 14(2), 179–211 (1990)CrossRefGoogle Scholar
  26. 26.
    Lin, T., Horne, B.G., Tiňo, P., Giles, C.L.: Learning long-term dependencies in narx recurrent neural networks. IEEE Trans. Neural Netw. 7(6), 1329–1338 (1996)CrossRefGoogle Scholar
  27. 27.
    Waibel, A., Hanazawa, T., Hinton, G., Shikano, K., Lang, K.J.: Phoneme recognition using time-delay neural networks. IEEE Trans. Acoust. Speech Signal Process. 37(3), 328–339 (1989)CrossRefGoogle Scholar
  28. 28.
    Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20, 37–46 (1960)Google Scholar
  29. 29.
    Spearman, C.E.: The proof and measurement of association between two things. Am. J. Psychol. 15, 88–103 (1904)Google Scholar
  30. 30.
    Cohen, J.: Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychol. Bull. 70(4), 213 (1968)CrossRefGoogle Scholar
  31. 31.
    De Jesús, O., Hagan, M.T.: Backpropagation algorithms for a broad class of dynamic networks. IEEE Trans. Neural Netw. 18(1), 14–27 (2007)CrossRefGoogle Scholar
  32. 32.
    De Jesús, O., Horn, J.M., Hagan, M.T.: Analysis of recurrent network training and suggestions for improvements. In: Proccedings of the International Joint Conference on Neural Networks (IJCNN’01) 2001. vol. 4, pp. 2632–2637. IEEE (2001)Google Scholar
  33. 33.
    Horn, J., De Jesús, O., Hagan, M.T.: Spurious valleys in the error surface of recurrent networksanalysis and avoidance. IEEE Trans. Neural Netw. 20(4), 686–700 (2009)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Singapore 2017

Authors and Affiliations

  1. 1.Institute of Communications EngineeringUlm UniversityUlmGermany
  2. 2.Engineering DepartmentCambridge UniversityCambridgeUK

Personalised recommendations