Affective Dialogue Management Using Factored POMDPs

  • Trung H. Bui
  • Job Zwiers
  • Mannes Poel
  • Anton Nijholt


Partially Observable Markov Decision Processes (POMDPs) have been demonstrated empirically to be good models for robust spoken dialogue design. This chapter shows that such models are also very appropriate for designing affective dialogue systems. We describe how to model affective dialogue systems using POMDPs and propose a novel approach to develop an affective dialogue model using factored POMDPs. We apply this model for a single-slot route navigation dialogue problem as a proof of concept. The experimental results demonstrate that integrating user’s affect into a POMDP-based dialogue manager is not only a nice idea but is also helpful for improving the dialogue manager performance given that the user’s affect influences their behavior. Further, our practical findings and experiments on the model tractability are expected to be helpful for designers and researchers who are interested in practical implementation of dialogue systems using the state-of-the-art POMDP techniques.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ai, H., Weng, F.: User simulation as testing for spoken dialog systems. In: Schlangen, D., Hockey, B.A. (eds.) Proceedings of the 9th SIGDial Workshop on Discourse and Dialogue (SIGdial 2008), Columbus, Ohio, USA, pp. 164–171 (2008)Google Scholar
  2. 2.
    André, E., Dybkjær, L., Minker, W., Heisterkamp, P. (eds.): ADS 2004. LNCS (LNAI), vol. 3068. Springer, Heidelberg (2004)Google Scholar
  3. 3.
    Astrom, K.J.: Optimal control of Markov processes with incomplete state information. Journal of Mathematical Analysis and Applications 10, 174–205 (1965)CrossRefMathSciNetGoogle Scholar
  4. 4.
    Ball, E.: A Bayesian heart: Computer recognition and simulation of emotion. In: Robert Trappl, P.P., Payr, S. (eds.) Emotions in Humans and Artifacts, vol. 11, pp. 303–332. The MIT Press, Cambridge (2003)Google Scholar
  5. 5.
    Batliner, A., Fischer, K., Huber, R., Spilker, J., Nöth, E.: How to find trouble in communication. Speech Communication 40(1-2), 117–143 (2003)MATHCrossRefGoogle Scholar
  6. 6.
    Bhatt, K., Argamon, S., Evens, M.: Hedged responses and expressions of affect in human/human and human/computer tutorial interactions. In: Forbus, K., Gentner, D., Regier, T. (eds.) Proceedings of the 26th Annual Conference of the Cognitive Science Society (CogSci 2004), Chicago, Illinois, USA, pp. 114–119 (2004)Google Scholar
  7. 7.
    Boutilier, C., Poole, D.: Computing optimal policies for partially observable decision processes using compact representations. In: Proceedings of the 13th National Conference on Artificial Intelligence (AAAI 1996), Portland, Oregon, USA, vol. 2, pp. 1168–1175 (1996)Google Scholar
  8. 8.
    Brooks, A., Makarenkoa, A., Williamsa, S., Durrant-Whytea, H.: Parametric POMDPs for planning in continuous state spaces. Robotics and Autonomous Systems 54(11), 887–897 (2006)CrossRefGoogle Scholar
  9. 9.
    Bui, T.H.: Multimodal dialogue management - State of the art. Tech. rep., University of Twente (2006)Google Scholar
  10. 10.
    Bui, T.H.: Toward affective dialogue management using partially observable markov decision processes. Ph.D. thesis, University of Twente (2008)Google Scholar
  11. 11.
    Bui, T.H., Poel, M., Nijholt, A., Zwiers, J.: A tractable hybrid DDN-POMDP approach to affective dialogue modeling for probabilistic frame-based dialogue systems. Natural Language Engineering 15(2), 273–307 (2009)CrossRefGoogle Scholar
  12. 12.
    Bui, T.H., Rajman, M., Melichar, M.: Rapid dialogue prototyping methodology. In: Sojka, P., Kopeček, I., Pala, K. (eds.) TSD 2004. LNCS (LNAI), vol. 3206, pp. 579–586. Springer, Heidelberg (2004)Google Scholar
  13. 13.
    Bui, T.H., van Schooten, B., Hofs, D.: Practical dialogue manager development using POMDPs. In: Keizer, S., Bunt, H., Paek, T. (eds.) Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue (SIGdial 2007), Antwerp, Belgium, pp. 215–218 (2007)Google Scholar
  14. 14.
    Bui, T.H., Zwiers, J., Nijholt, A., Poel, M.: Generic dialogue modeling for multi-application dialogue systems. In: Renals, S., Bengio, S. (eds.) MLMI 2005. LNCS, vol. 3869, pp. 174–186. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  15. 15.
    Eckert, W., Levin, E., Pieraccini, R.: User modelling for spoken dialogue system evaluation. In: Prococeedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU 1997), pp. 80–87. IEEE, Santa Barbara (1997)CrossRefGoogle Scholar
  16. 16.
    Hauskrecht, M.: Value-function approximations for partially observable Markov decision processes. Journal of Artificial Intelligence Research (JAIR) 13, 33–94 (2000)MATHMathSciNetGoogle Scholar
  17. 17.
    Heylen, D., Nijholt, A., op den Akker, R.: Affect in tutoring dialogues. Applied Artificial Intelligence 19, 287–311 (2005)CrossRefGoogle Scholar
  18. 18.
    Hoey, J., von Bertoldi, A., Poupart, P., Mihailidis, A.: Assisting persons with dementia during handwashing using a partially observable markov decision process. In: Proceedings of the 5th International Conference on Vision Systems (ICVS 2007), Bielefeld, Germany (2007)Google Scholar
  19. 19.
    Howard, R.A.: Dynamic Programming and Markov Process. The MIT Press, Cambridge (1960)Google Scholar
  20. 20.
    Jurafsky, D., Martin, J.: Speech and Language Processing: An Introduction to Natural Language Processing. Prentice Hall, Englewood Cliffs (2000)Google Scholar
  21. 21.
    Kaelbling, L.P., Littman, M.L., Cassandra, A.R.: Planning and acting in partially observable stochastic domains. Artificial Intelligence 101(1-2), 99–134 (1998)MATHCrossRefMathSciNetGoogle Scholar
  22. 22.
    Levelt, W.J.: Speaking: From Intention to Articulation. The MIT Press, Cambridge (1989)Google Scholar
  23. 23.
    Levin, E., Pieraccini, R., Eckert, W.: A stochastic model of human-machine interaction for learning dialogue strategies. IEEE Transactions on Speech and Audio Processing 8(1), 11–23 (2000)CrossRefGoogle Scholar
  24. 24.
    Littman, M.L., Cassandra, A.R., Kaelbling, L.P.: Learning policies for partially observable environments: Scaling up. In: Prieditis, A., Russell, S.J. (eds.) Proceedings of the 12th International Conference on Machine Learning (ICML 1995), pp. 362–370. Morgan Kaufmann, Tahoe City (1995)Google Scholar
  25. 25.
    Martinovsky, B., Traum, D.R.: The error is the clue: Breakdown in human-machine interaction. In: Proceedings of the ISCA Tutorial and Research Workshop on Error handling in Spoken Dialogue Systems (EHSD 2003), Château d’Oex, Vaud, Switzerland, pp. 11–16 (2003)Google Scholar
  26. 26.
    McTear, M.: Spoken dialogue technology: Enabling the conversational user interface. ACM Computing Survey 34(1) (2002)Google Scholar
  27. 27.
    Monahan, G.E.: A survey of partially observable markov decision processes: Theory, models, and algorithms. Management Science 28-1, 1–16 (1982)CrossRefMathSciNetGoogle Scholar
  28. 28.
    Ortony, A., Clore, G.L., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1988)Google Scholar
  29. 29.
    Paek, T., Horvitz, E.: Conversation as action under uncertainty. In: Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence (UAI 2000), pp. 455–464. Morgan Kaufmann, San Francisco (2000)Google Scholar
  30. 30.
    Picard, R.W.: Affective Computing. The MIT Press, Cambridge (1997)Google Scholar
  31. 31.
    Pietquin, O.: A framework for unsupervised learning of dialogue strategies. Ph.D. thesis, Universitaires de Louvain (2004)Google Scholar
  32. 32.
    Puterman, M.L.: Markov decision processes. In: Heyman, D., Sobel, M. (eds.) Handbook in Operations Research and Management Science, vol. 2, pp. 331–434. Elsevier, Amsterdam (1990)Google Scholar
  33. 33.
    de Rosis, F., Novielli, N., Carofiglio, V., Cavalluzzi, A., Carolis, B.D.: User modeling and adaptation in health promotion dialogs with an animated character. Journal of Biomedical Informatics 39(5), 514–531 (2006)CrossRefGoogle Scholar
  34. 34.
    Roy, N., Pineau, J., Thrun, S.: Spoken dialogue management using probabilistic reasoning. In: Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics (ACL 2000), pp. 93–100. ACL, Hong Kong (2000)CrossRefGoogle Scholar
  35. 35.
    Roy, N., Thrun, S.: Coastal navigation with mobile robots. In: Solla, S.A., Leen, T.K., Müller, K.R. (eds.) Advances in Neural Information Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29 - December 4, 1999], pp. 1043–1049. The MIT Press, Denver (2000)Google Scholar
  36. 36.
    Schatzmann, J., Thomson, B., Weilhammer, K., Ye, H., Young, S.: Agenda-based user simulation for bootstrapping a POMDP dialogue system. In: Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2007), pp. 149–152. ACL, Rochester (2007)Google Scholar
  37. 37.
    Schatzmann, J., Weilhammer, K., Stuttle, M., Young, S.: A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. Knowledge Engineering Review 21(2), 97–126 (2006)CrossRefGoogle Scholar
  38. 38.
    Scheffler, K., Young, S.J.: Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning. In: Marcus, M. (ed.) Proceedings of the 2nd International Conference on Human Language Technology Research (HLT 2002), pp. 12–18. Morgan Kaufmann, San Francisco (2002)Google Scholar
  39. 39.
    Smallwood, R.D., Sondik, E.J.: The optimal control of partially observable Markov processes over a finite horizon. Operations Research 21-5, 1071–1088 (1973)CrossRefGoogle Scholar
  40. 40.
    Sondik, E.J.: The optimal control of partially observable markov decision processes. Ph.D. thesis, Stanford University (1971)Google Scholar
  41. 41.
    Spaan, M.T.J., Vlassis, N.: Perseus: Randomized point-based value iteration for POMDPs. Journal of Artificial Intelligence Research (JAIR) 24, 195–220 (2005)MATHGoogle Scholar
  42. 42.
    Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. The MIT Press, Cambridge (1998)Google Scholar
  43. 43.
    Traum, D., Larsson, S.: The information state approach to dialogue management. In: van Kuppevelt, J., Smith, R.W. (eds.) Current and New Directions in Discourse and Dialogue, ch. 15, pp. 325–353. Kluwer Academic Publishers, Dordrecht (2003)Google Scholar
  44. 44.
    Williams, J., Poupart, P., Young, S.: Partially Observable Markov Decision Processes with Continuous Observations for Dialog Management. In: Williams, J., Poupart, P., Young, S. (eds.) Recent Trends in Discourse and Dialogue, chap, pp. 191–217. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  45. 45.
    Williams, J., Young, S.: Scaling up POMDPs for dialogue management: the summary POMDP method. In: Proceedings of the IEEE workshop on Automatic Speech Recognition and Understanding (ASRU 2005), Cancún, Mexico, pp. 250–255 (2005)Google Scholar
  46. 46.
    Williams, J.D.: Partially observable Markov decision processes for dialog management. Ph.D. thesis, Cambridge University (2006)Google Scholar
  47. 47.
    Williams, J.D., Poupart, P., Young, S.: Factored partially observable Markov decision processes for dialogue management. In: Zukerman, I., Alexandersson, J., Jönsson, A. (eds.) Proceedings of the 4th Workshop on Knowledge and Reasoning in Practical Dialog Systems (KRPD 2005), Edinburgh, Scotland, pp. 76–82 (2005)Google Scholar
  48. 48.
    Williams, J.D., Poupart, P., Young, S.: Partially observable Markov decision processes with continuous observations for dialogue management. In: Proceedings of the 6th SigDial Workshop on Discourse and Dialogue, SIGdial 2005 (2005)Google Scholar
  49. 49.
    Williams, J.D., Young, S.: Partially observable markov decision processes for spoken dialog systems. Computer Speech and Language 21(2), 393–422 (2007)CrossRefGoogle Scholar
  50. 50.
    Young, S.: Talking to machines (statistically speaking). In: Proceedings of the 7th International Conference on Spoken Language Processing (ICSLP 2002), Denver, Colorado, USA, pp. 9–16 (2002)Google Scholar
  51. 51.
    Young, S., Gasić, M., Keizer, S., Mairesse, F., Schatzmann, J., Thomson, B., Yu, K.: The hidden information state model: A practical framework for pomdp-based spoken dialogue management. Computer Speech and Language (2009)Google Scholar
  52. 52.
    Zhang, B., Cai, Q., Mao, J., Guo, B.: Spoken dialog management as planning and acting under uncertainty. In: Proceedings of the 7th European Conference on Speech Communication and Technology (EUROSPEECH 2001), Aalborg, Denmark, pp. 2169–2172 (2001)Google Scholar
  53. 53.
    Zhang, N.L.: Efficient planning in stochastic domains through exploiting problem characteristics. Tech. Rep. HKUST-CS95-40, Hong Kong University of Science and Technology (1995)Google Scholar
  54. 54.
    Zhang, N.L., Zhang, W.: Speeding up the convergence of value iteration in partially observable markov decision processes. Journal of Artificial Intelligence Research (JAIR) 14, 29–51 (2001)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Trung H. Bui
    • 1
  • Job Zwiers
    • 2
  • Mannes Poel
    • 2
  • Anton Nijholt
    • 2
  1. 1.Center for the Study of Language and InformationStanford UniversityStanfordUSA
  2. 2.Human Media Interaction GroupUniversity of TwenteEnschedeThe Netherlands

Personalised recommendations