Skip to main content
SpringerLink
Log in

Affective Dialogue Management Using Factored POMDPs

  • Chapter
Interactive Collaborative Information Systems

Part of the book series: Studies in Computational Intelligence ((SCI,volume 281))

Abstract

Partially Observable Markov Decision Processes (POMDPs) have been demonstrated empirically to be good models for robust spoken dialogue design. This chapter shows that such models are also very appropriate for designing affective dialogue systems. We describe how to model affective dialogue systems using POMDPs and propose a novel approach to develop an affective dialogue model using factored POMDPs. We apply this model for a single-slot route navigation dialogue problem as a proof of concept. The experimental results demonstrate that integrating user’s affect into a POMDP-based dialogue manager is not only a nice idea but is also helpful for improving the dialogue manager performance given that the user’s affect influences their behavior. Further, our practical findings and experiments on the model tractability are expected to be helpful for designers and researchers who are interested in practical implementation of dialogue systems using the state-of-the-art POMDP techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ai, H., Weng, F.: User simulation as testing for spoken dialog systems. In: Schlangen, D., Hockey, B.A. (eds.) Proceedings of the 9th SIGDial Workshop on Discourse and Dialogue (SIGdial 2008), Columbus, Ohio, USA, pp. 164–171 (2008)

    Google Scholar 

  2. André, E., Dybkjær, L., Minker, W., Heisterkamp, P. (eds.): ADS 2004. LNCS (LNAI), vol. 3068. Springer, Heidelberg (2004)

    Google Scholar 

  3. Astrom, K.J.: Optimal control of Markov processes with incomplete state information. Journal of Mathematical Analysis and Applications 10, 174–205 (1965)

    Article  MathSciNet  Google Scholar 

  4. Ball, E.: A Bayesian heart: Computer recognition and simulation of emotion. In: Robert Trappl, P.P., Payr, S. (eds.) Emotions in Humans and Artifacts, vol. 11, pp. 303–332. The MIT Press, Cambridge (2003)

    Google Scholar 

  5. Batliner, A., Fischer, K., Huber, R., Spilker, J., Nöth, E.: How to find trouble in communication. Speech Communication 40(1-2), 117–143 (2003)

    Article  MATH  Google Scholar 

  6. Bhatt, K., Argamon, S., Evens, M.: Hedged responses and expressions of affect in human/human and human/computer tutorial interactions. In: Forbus, K., Gentner, D., Regier, T. (eds.) Proceedings of the 26th Annual Conference of the Cognitive Science Society (CogSci 2004), Chicago, Illinois, USA, pp. 114–119 (2004)

    Google Scholar 

  7. Boutilier, C., Poole, D.: Computing optimal policies for partially observable decision processes using compact representations. In: Proceedings of the 13th National Conference on Artificial Intelligence (AAAI 1996), Portland, Oregon, USA, vol. 2, pp. 1168–1175 (1996)

    Google Scholar 

  8. Brooks, A., Makarenkoa, A., Williamsa, S., Durrant-Whytea, H.: Parametric POMDPs for planning in continuous state spaces. Robotics and Autonomous Systems 54(11), 887–897 (2006)

    Article  Google Scholar 

  9. Bui, T.H.: Multimodal dialogue management - State of the art. Tech. rep., University of Twente (2006)

    Google Scholar 

  10. Bui, T.H.: Toward affective dialogue management using partially observable markov decision processes. Ph.D. thesis, University of Twente (2008)

    Google Scholar 

  11. Bui, T.H., Poel, M., Nijholt, A., Zwiers, J.: A tractable hybrid DDN-POMDP approach to affective dialogue modeling for probabilistic frame-based dialogue systems. Natural Language Engineering 15(2), 273–307 (2009)

    Article  Google Scholar 

  12. Bui, T.H., Rajman, M., Melichar, M.: Rapid dialogue prototyping methodology. In: Sojka, P., Kopeček, I., Pala, K. (eds.) TSD 2004. LNCS (LNAI), vol. 3206, pp. 579–586. Springer, Heidelberg (2004)

    Google Scholar 

  13. Bui, T.H., van Schooten, B., Hofs, D.: Practical dialogue manager development using POMDPs. In: Keizer, S., Bunt, H., Paek, T. (eds.) Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue (SIGdial 2007), Antwerp, Belgium, pp. 215–218 (2007)

    Google Scholar 

  14. Bui, T.H., Zwiers, J., Nijholt, A., Poel, M.: Generic dialogue modeling for multi-application dialogue systems. In: Renals, S., Bengio, S. (eds.) MLMI 2005. LNCS, vol. 3869, pp. 174–186. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  15. Eckert, W., Levin, E., Pieraccini, R.: User modelling for spoken dialogue system evaluation. In: Prococeedings of the IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU 1997), pp. 80–87. IEEE, Santa Barbara (1997)

    Chapter  Google Scholar 

  16. Hauskrecht, M.: Value-function approximations for partially observable Markov decision processes. Journal of Artificial Intelligence Research (JAIR) 13, 33–94 (2000)

    MATH  MathSciNet  Google Scholar 

  17. Heylen, D., Nijholt, A., op den Akker, R.: Affect in tutoring dialogues. Applied Artificial Intelligence 19, 287–311 (2005)

    Article  Google Scholar 

  18. Hoey, J., von Bertoldi, A., Poupart, P., Mihailidis, A.: Assisting persons with dementia during handwashing using a partially observable markov decision process. In: Proceedings of the 5th International Conference on Vision Systems (ICVS 2007), Bielefeld, Germany (2007)

    Google Scholar 

  19. Howard, R.A.: Dynamic Programming and Markov Process. The MIT Press, Cambridge (1960)

    Google Scholar 

  20. Jurafsky, D., Martin, J.: Speech and Language Processing: An Introduction to Natural Language Processing. Prentice Hall, Englewood Cliffs (2000)

    Google Scholar 

  21. Kaelbling, L.P., Littman, M.L., Cassandra, A.R.: Planning and acting in partially observable stochastic domains. Artificial Intelligence 101(1-2), 99–134 (1998)

    Article  MATH  MathSciNet  Google Scholar 

  22. Levelt, W.J.: Speaking: From Intention to Articulation. The MIT Press, Cambridge (1989)

    Google Scholar 

  23. Levin, E., Pieraccini, R., Eckert, W.: A stochastic model of human-machine interaction for learning dialogue strategies. IEEE Transactions on Speech and Audio Processing 8(1), 11–23 (2000)

    Article  Google Scholar 

  24. Littman, M.L., Cassandra, A.R., Kaelbling, L.P.: Learning policies for partially observable environments: Scaling up. In: Prieditis, A., Russell, S.J. (eds.) Proceedings of the 12th International Conference on Machine Learning (ICML 1995), pp. 362–370. Morgan Kaufmann, Tahoe City (1995)

    Google Scholar 

  25. Martinovsky, B., Traum, D.R.: The error is the clue: Breakdown in human-machine interaction. In: Proceedings of the ISCA Tutorial and Research Workshop on Error handling in Spoken Dialogue Systems (EHSD 2003), Château d’Oex, Vaud, Switzerland, pp. 11–16 (2003)

    Google Scholar 

  26. McTear, M.: Spoken dialogue technology: Enabling the conversational user interface. ACM Computing Survey 34(1) (2002)

    Google Scholar 

  27. Monahan, G.E.: A survey of partially observable markov decision processes: Theory, models, and algorithms. Management Science 28-1, 1–16 (1982)

    Article  MathSciNet  Google Scholar 

  28. Ortony, A., Clore, G.L., Collins, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1988)

    Google Scholar 

  29. Paek, T., Horvitz, E.: Conversation as action under uncertainty. In: Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence (UAI 2000), pp. 455–464. Morgan Kaufmann, San Francisco (2000)

    Google Scholar 

  30. Picard, R.W.: Affective Computing. The MIT Press, Cambridge (1997)

    Google Scholar 

  31. Pietquin, O.: A framework for unsupervised learning of dialogue strategies. Ph.D. thesis, Universitaires de Louvain (2004)

    Google Scholar 

  32. Puterman, M.L.: Markov decision processes. In: Heyman, D., Sobel, M. (eds.) Handbook in Operations Research and Management Science, vol. 2, pp. 331–434. Elsevier, Amsterdam (1990)

    Google Scholar 

  33. de Rosis, F., Novielli, N., Carofiglio, V., Cavalluzzi, A., Carolis, B.D.: User modeling and adaptation in health promotion dialogs with an animated character. Journal of Biomedical Informatics 39(5), 514–531 (2006)

    Article  Google Scholar 

  34. Roy, N., Pineau, J., Thrun, S.: Spoken dialogue management using probabilistic reasoning. In: Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics (ACL 2000), pp. 93–100. ACL, Hong Kong (2000)

    Chapter  Google Scholar 

  35. Roy, N., Thrun, S.: Coastal navigation with mobile robots. In: Solla, S.A., Leen, T.K., Müller, K.R. (eds.) Advances in Neural Information Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29 - December 4, 1999], pp. 1043–1049. The MIT Press, Denver (2000)

    Google Scholar 

  36. Schatzmann, J., Thomson, B., Weilhammer, K., Ye, H., Young, S.: Agenda-based user simulation for bootstrapping a POMDP dialogue system. In: Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2007), pp. 149–152. ACL, Rochester (2007)

    Google Scholar 

  37. Schatzmann, J., Weilhammer, K., Stuttle, M., Young, S.: A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. Knowledge Engineering Review 21(2), 97–126 (2006)

    Article  Google Scholar 

  38. Scheffler, K., Young, S.J.: Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning. In: Marcus, M. (ed.) Proceedings of the 2nd International Conference on Human Language Technology Research (HLT 2002), pp. 12–18. Morgan Kaufmann, San Francisco (2002)

    Google Scholar 

  39. Smallwood, R.D., Sondik, E.J.: The optimal control of partially observable Markov processes over a finite horizon. Operations Research 21-5, 1071–1088 (1973)

    Article  Google Scholar 

  40. Sondik, E.J.: The optimal control of partially observable markov decision processes. Ph.D. thesis, Stanford University (1971)

    Google Scholar 

  41. Spaan, M.T.J., Vlassis, N.: Perseus: Randomized point-based value iteration for POMDPs. Journal of Artificial Intelligence Research (JAIR) 24, 195–220 (2005)

    MATH  Google Scholar 

  42. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. The MIT Press, Cambridge (1998)

    Google Scholar 

  43. Traum, D., Larsson, S.: The information state approach to dialogue management. In: van Kuppevelt, J., Smith, R.W. (eds.) Current and New Directions in Discourse and Dialogue, ch. 15, pp. 325–353. Kluwer Academic Publishers, Dordrecht (2003)

    Google Scholar 

  44. Williams, J., Poupart, P., Young, S.: Partially Observable Markov Decision Processes with Continuous Observations for Dialog Management. In: Williams, J., Poupart, P., Young, S. (eds.) Recent Trends in Discourse and Dialogue, chap, pp. 191–217. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  45. Williams, J., Young, S.: Scaling up POMDPs for dialogue management: the summary POMDP method. In: Proceedings of the IEEE workshop on Automatic Speech Recognition and Understanding (ASRU 2005), Cancún, Mexico, pp. 250–255 (2005)

    Google Scholar 

  46. Williams, J.D.: Partially observable Markov decision processes for dialog management. Ph.D. thesis, Cambridge University (2006)

    Google Scholar 

  47. Williams, J.D., Poupart, P., Young, S.: Factored partially observable Markov decision processes for dialogue management. In: Zukerman, I., Alexandersson, J., Jönsson, A. (eds.) Proceedings of the 4th Workshop on Knowledge and Reasoning in Practical Dialog Systems (KRPD 2005), Edinburgh, Scotland, pp. 76–82 (2005)

    Google Scholar 

  48. Williams, J.D., Poupart, P., Young, S.: Partially observable Markov decision processes with continuous observations for dialogue management. In: Proceedings of the 6th SigDial Workshop on Discourse and Dialogue, SIGdial 2005 (2005)

    Google Scholar 

  49. Williams, J.D., Young, S.: Partially observable markov decision processes for spoken dialog systems. Computer Speech and Language 21(2), 393–422 (2007)

    Article  Google Scholar 

  50. Young, S.: Talking to machines (statistically speaking). In: Proceedings of the 7th International Conference on Spoken Language Processing (ICSLP 2002), Denver, Colorado, USA, pp. 9–16 (2002)

    Google Scholar 

  51. Young, S., Gasić, M., Keizer, S., Mairesse, F., Schatzmann, J., Thomson, B., Yu, K.: The hidden information state model: A practical framework for pomdp-based spoken dialogue management. Computer Speech and Language (2009)

    Google Scholar 

  52. Zhang, B., Cai, Q., Mao, J., Guo, B.: Spoken dialog management as planning and acting under uncertainty. In: Proceedings of the 7th European Conference on Speech Communication and Technology (EUROSPEECH 2001), Aalborg, Denmark, pp. 2169–2172 (2001)

    Google Scholar 

  53. Zhang, N.L.: Efficient planning in stochastic domains through exploiting problem characteristics. Tech. Rep. HKUST-CS95-40, Hong Kong University of Science and Technology (1995)

    Google Scholar 

  54. Zhang, N.L., Zhang, W.: Speeding up the convergence of value iteration in partially observable markov decision processes. Journal of Artificial Intelligence Research (JAIR) 14, 29–51 (2001)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Center for the Study of Language and Information, Stanford University, 210 Panama St, Stanford, CA, 94305, USA

    Trung H. Bui

  2. Human Media Interaction Group, University of Twente, Postbus 217, 7500, AE, Enschede, The Netherlands

    Job Zwiers, Mannes Poel & Anton Nijholt

Authors
  1. Trung H. Bui
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Job Zwiers
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mannes Poel
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anton Nijholt
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Delft Center for Systems and Control, Delft University of Technology, Mekelweg 2, 2628, Delft, CD, The Netherlands

    Robert Babuška

  2. Faculty of Science, Informatics Institute, Science Park 107, 1098, Amsterdam, XG, The Netherlands

    Frans C. A. Groen

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Bui, T.H., Zwiers, J., Poel, M., Nijholt, A. (2010). Affective Dialogue Management Using Factored POMDPs. In: Babuška, R., Groen, F.C.A. (eds) Interactive Collaborative Information Systems. Studies in Computational Intelligence, vol 281. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-11688-9_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-11688-9_8

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-11687-2

  • Online ISBN: 978-3-642-11688-9

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation