A User Model for Dialog System Evaluation Based on Activation of Subgoals

Conference paper


User models have become increasingly popular to conduct simulation-based testing of spoken dialog systems. These models usually describe users’ overt behavior, as opposed to the underlying reasons for the observed actions. While such models are useful to generate test data, a causal model might be more generally applicable to different systems and, in addition, allows to derive useful information for data analysis and prediction of user judgments. Thus, a modeling approach trying to explain user behavior is proposed in this paper, which is based on Dörner’s PSI theory. The evaluation shows that the utterances generated by this model are similar to those of real users.


User Model User Behavior Automatic Speech Recognition Real User Dialog State 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Blackmon, M., Kitajima, M., Polson, P.: Tool for accurately predicting website navigation problems, non-problems, problem severity, and effectiveness of repairs. In: Proceedings of CHI ’05, pp. 31–40 (2005)Google Scholar
  2. 2.
    Branigan, H., Pickering, M., Pearson, J., McLean, J.: Linguistic alignment between humans and computers. J. Pragmat. 42, 2355–2368 (2010)CrossRefGoogle Scholar
  3. 3.
    Chung, G.: Developing a flexible spoken dialog system using simulation. In: Proceedings of ACL 2004, pp. 93–98 (2004)Google Scholar
  4. 4.
    Dörner, D.: Die Mechanik des Seelenwagens. Eine neuronale Theorie der Handlungsregulation, 1st Edition, Verlag Hans Huber, Bern (2002)Google Scholar
  5. 5.
    Eckert, W., Levin, E., Pieraccini, R.: User modeling for spoken dialogue system evaluation. In: Proceedings of the IEEE Workshop on Automatic Speech Recognition and Understanding, pp. 80–87 (1997)Google Scholar
  6. 6.
    Engelbrecht, K.-P.: Estimating spoken dialog system quality with user models. Ph.D. Thesis, Berlin Institute of Technology, Berlin (2012)Google Scholar
  7. 7.
    Engelbrecht, K.-P., Möller, S.: Correlation between model-based approximations of grounding-related cognition and user judgments. In: Proceedings of Interspeech 2012 (2012)Google Scholar
  8. 8.
    López-Cózar, R., Callejas, Z., McTear, M.: Testing the performance of spoken dialogue systems by means of an artificially simulated user. Artif. Intell. Rev. 26, 291–323 (2006)CrossRefGoogle Scholar
  9. 9.
    Möller, S.: Quality of Telephone-Based Spoken Dialog Systems. Springer, New York (2005)Google Scholar
  10. 10.
    Möller, S., Schleicher, R., Butenkov, D., Engelbrecht, K.-P., Gödde, F., Scheffler, T., Roller, R., Reithinger, N.: Usability engineering for spoken dialogue systems via statistical user models. In: Proceedings of IWSDS ’09 (2009)Google Scholar
  11. 11.
    Pietquin, O.: A framework for unsupervised learning of dialogue strategies. Ph.D. Thesis, TCTS Lab, Mons (2004)Google Scholar
  12. 12.
    Schatzmann, J., Georgila, K., Young., S.: Quantitative evaluation of user simulation techniques for spoken dialogue systems. In: Proceedings of SIGDial, pp. 45–54 (2005)Google Scholar
  13. 13.
    Schatzmann, J., Thomson, B., Weilhammer, K., Ye, H., Young, S.: Agenda-based user simulation for bootstrapping a pomdp dialogue system. In: Proceedings of HLT/NAACL, pp. 149–152 (2007)Google Scholar
  14. 14.
    Walker, M., Litman, D., Kamm, C., Abella, A.: Paradise: A framework for evaluating spoken dialogue agents. In: Proceedings of ACL/EACL, pp. 271–280 (1997)Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  1. 1.Quality and Usability Lab, Telekom Innovation LaboratoriesTU BerlinBerlinGermany

Personalised recommendations