Modeling Brief Alcohol Intervention Dialogue with MDPs for Delivery by ECAs

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8108)


This paper describes the design of a multimodal spoken dialogue system using Markov Decision Processes (MDPs) to enable embodied conversational virtual health coach agents to deliver brief interventions for lifestyle behavior change - in particular excessive alcohol consumption. Its contribution is two fold. First, it is the first attempt to-date to study stochastic dialogue policy optimization techniques in the health dialogue domain. Second, it provides a model for longer branching dialogues (in terms of number of dialogue turns and number of slots) than the usual slot filling dialogue interactions currently available (e.g. tourist information domain). In addition, the model forms the basis for the generation of a richly annotated dialogue corpus, which is essential for applying optimization methods based on reinforcement learning.


spoken dialogue system markov decision Processes reinforcement learning embodied conversational agent (ECA) intelligent virtual agents brief intervention behavior change alcoholism at-risk drinking 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Mokdad, A.H., Marks, J.S., Stroup, D.F., Gerberding, J.L.: Actual causes of death in the united states, 2000. JAMA 291(10), 1238–1245 (2000)CrossRefGoogle Scholar
  2. 2.
    Bouchery, E.E., Harwood, H.J., Sacks, J.J., Simon, C.J., Brewer, R.D.: Economic costs of excessive alcohol consumption in the U.S., 2006. American Journal of Preventive Medicine 41(5), 516–524 (2011)CrossRefGoogle Scholar
  3. 3.
    Moyer, A., Finney, J.W., Swearingen, C.E., Vergun, P.: Brief interventions for alcohol problems: a meta-analytic review of controlled investigations in treatment-seeking and non-treatment-seeking populations. Addiction 97(3), 279–292 (2002)CrossRefGoogle Scholar
  4. 4.
    Miller, W.R., Wilbourne, P.L.: Mesa grande: a methodological analysis of clinical trials of treatments for alcohol use disorders. Addiction 97(3), 265–277 (2002)CrossRefGoogle Scholar
  5. 5.
    Babor, T.F., Grant, M.: Programme on substance abuse: project on identification and management of alcohol-related problems. Report on phase ii, an randomized clinical trial of brief interventions in primary health care (1992)Google Scholar
  6. 6.
    NIAAA: NIAAA Alcohol Alert No. 66: Brief interventions (2006)Google Scholar
  7. 7.
    Riper, H., Spek, V., Boon, B., Conijn, B., Kramer, J., Martin-Abello, K., Smit, F.: Effectiveness of e-self-help interventions for curbing adult problem drinking: A meta-analysis. J. Med. Internet Res. 13(2), e42 (2011)Google Scholar
  8. 8.
    White, A., Kavanagh, D., Stallman, H., Klein, B., Kay-Lambkin, F., Proudfoot, J., Drennan, J., Connor, J., Baker, A., Hines, E., Young, R.: Online alcohol interventions: A systematic review. J. Med. Internet Res. 12(5), e62 (2010)Google Scholar
  9. 9.
    Portnoy, D.B., Scott-Sheldon, L.A.J., Johnson, B.T., Carey, M.P.: Computer-delivered interventions for health promotion and behavioral risk reduction: a meta-analysis of 75 randomized controlled trials, 1988-2007. Preventive Medicine 47(1), 3–16 (2008)CrossRefGoogle Scholar
  10. 10.
    Hester, R.K., Squires, D.D., Delaney, H.D.: The Drinker’s Check-up: 12-month outcomes of a controlled clinical trial of a stand-alone software program for problem drinkers. Journal of Substance Abuse Treatment 28(2), 159–169 (2005)CrossRefGoogle Scholar
  11. 11.
    Mauriello, L.M., Gökbayrak, N.S., Marter, D.F.V., Paiva, A.L., Prochaska, J.M.: An Internet-Based Computer-Tailored Intervention to Promote Responsible Drinking: Findings from a Pilot Test with Employed Adults. Alcoholism Treatment 30(1), 1–15 (2011)Google Scholar
  12. 12.
    Lisetti, C., Yasavur, U., Leon, C.D., Amini, R., Rishe, N., Visser, U.: Building an On-demand Avatar-based Health Intervention for Behavior Change. FLAIRS (2012)Google Scholar
  13. 13.
    Cassell, J., Nakano, Y.I., Bickmore, T.W., Sidner, C.L., Rich, C.: Non-verbal cues for discourse structure. In: Proceedings of the 39th Annual Meeting on Association for Computational Linguistics, ACL 2001, pp. 114–123 (2001)Google Scholar
  14. 14.
    Moridis, C.N., Economides, A.A.: Affective Learning: Empathetic Agents with Emotional Facial and Tone of Voice Expressions. IEEE Transactions on Affective Computing 3(3), 260–272 (2012)CrossRefGoogle Scholar
  15. 15.
    Lisetti, C., Yasavur, U., Leon, C.D., Amini, R.: Building an On-Demand Avatar-Based Health Intervention for Behavior Change. In: Proceedings of the Twenty-Fifth International Florida Artificial Intelligence Research Society Conference (2012)Google Scholar
  16. 16.
    Schulman, D., Bickmore, T.W., Sidner, C.L.: An Intelligent Conversational Agent for Promoting Long-Term Health Behavior Change using Motivational Interviewing. In: Association for the Advancement of Artificial Intelligence (AAAI) Spring Symposium Series, pp. 61–64. Association for the Advancement of Artificial Intelligence (2011),
  17. 17.
    Morbini, F., Forbell, E., DeVault, D., Sagae, K., Traum, D.R., Rizzo, A.A.: A mixed-initiative conversational dialogue system for healthcare. In: Proceedings of the 13th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL 2012, pp. 137–139. Association for Computational Linguistics, Stroudsburg (2012)Google Scholar
  18. 18.
    Wang, N., Gratch, J.: Don’t just stare at me. In: CHI, Atlanta, GA, USA, pp. 1241–1249 (2010)Google Scholar
  19. 19.
    Kang, S.H., Gratch, J., Wang, N., Watt, J.: Does the contingency of agents’ nonverbal feedback affect users’ social anxiety? In: Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, vol. 1, pp. 120–127. International Foundation for Autonomous Agents and Multiagent Systems (2008)Google Scholar
  20. 20.
    NIAAA: Helping Patients Who Drink Too Much, A Clinician’s Guide (2007)Google Scholar
  21. 21.
    Bickmore, T.W., Schulman, D., Sidner, C.L.: A reusable framework for health counseling dialogue systems based on a behavioral medicine ontology. Journal of Biomedical Informatics 44(2), 183–197 (2011)Google Scholar
  22. 22.
    Ferguson, G., Quinn, J., Horwitz, C., Swift, M., Allen, J., Galescu, L.: Towards a Personal Health Management Assistant. Journal of Biomedical Informatics 43(5 suppl.), S13–S16 (2010)Google Scholar
  23. 23.
    Bickmore, T.W., Puskar, K., Schlenk, E.A., Pfeifer, L.M., Sereika, S.M.: Maintaining reality: Relational agents for antipsychotic medication adherence. Interacting with Computers 22(4), 276–288 (2010)CrossRefGoogle Scholar
  24. 24.
    Ferguson, G., Allen, J., et al.: Trips: An integrated intelligent problem-solving assistant. In: Proceedings of the National Conference on Artificial Intelligence, pp. 567–573. John Wiley & Sons Ltd. (1998)Google Scholar
  25. 25.
    Bohus, D., Rudnicky, A.I.: The ravenclaw dialog management framework: Architecture and systems. Computer Speech & Language 23(3), 332–361 (2009)CrossRefGoogle Scholar
  26. 26.
    Traum, D., Larsson, S.: The Information State Approach to Dialogue Management, pp. 325–353. Kluwer Academic Publishers (2003)Google Scholar
  27. 27.
    Bos, J., Klein, E., Lemon, O., Oka, T.: Dipper: Description and formalisation of an information-state update dialogue system architecture. In: 4th SIGdial Workshop on Discourse and Dialogue, pp. 115–124 (2003)Google Scholar
  28. 28.
    Sutton, R.S., Barto, A.G.: Reinforcement learning: An introduction, vol. 1. Cambridge Univ. Press (1998)Google Scholar
  29. 29.
    Lemon, O., Pietquin, O., et al.: Machine learning for spoken dialogue systems. In: Proceedings of the European Conference on Speech Communication and Technologies (Interspeech 2007), pp. 2685–2688 (2007)Google Scholar
  30. 30.
    Singh, S., Litman, D., Kearns, M., Walker, M.: Optimizing dialogue management with reinforcement learning: Experiments with the NJFun system. Journal of Artificial Intelligence Research 16, 105–133 (2002)Google Scholar
  31. 31.
    Young, S., Gašić, M., Thomson, B., Williams, J.: Pomdp-based statistical spoken dialog systems: A review (2013)Google Scholar
  32. 32.
    Papangelis, A., Kouroupas, N., Karkaletsis, V., Makedon, F.: An adaptive dialogue system with online dialogue policy learning. In: Maglogiannis, I., Plagianakos, V., Vlahavas, I. (eds.) SETN 2012. LNCS, vol. 7297, pp. 323–330. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  33. 33.
    Frampton, M., Lemon, O.: Recent research advances in reinforcement learning in spoken dialogue systems. Knowledge Engineering Review 24(4), 375–408 (2009)CrossRefGoogle Scholar
  34. 34.
    Young, S., Gašić, M., Keizer, S., Mairesse, F., Schatzmann, J., Thomson, B., Yu, K.: The hidden information state model: A practical framework for pomdp-based spoken dialogue management. Computer Speech & Language 24(2), 150–174 (2010)CrossRefGoogle Scholar
  35. 35.
    Chi, M., VanLehn, K., Litman, D.: Do micro-level tutorial decisions matter: Applying reinforcement learning to induce pedagogical tutorial tactics. In: Aleven, V., Kay, J., Mostow, J. (eds.) ITS 2010, Part I. LNCS, vol. 6094, pp. 224–234. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  36. 36.
    Walker, M.A., Passonneau, R., Boland, J.E.: Quantitative and qualitative evaluation of DARPA communicator spoken dialogue systems. In: Proceedings of the 39th Annual Meeting on Association for Computational Linguistics, ACL 2001, pp. 515–522. Association for Computational Linguistics, Stroudsburg (2001)Google Scholar
  37. 37.
    Georgila, K., Lemon, O., Henderson, J.: Automatic annotation of communicator dialogue data for learning dialogue strategies and user simulations. In: Ninth Workshop on the Semantics and Pragmatics of Dialogue (SEMDIAL: DIALOR). Citeseer (2005)Google Scholar
  38. 38.
    Frampton, M., Lemon, O.: Learning more effective dialogue strategies using limited dialogue move features. In: Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics, pp. 185–192. Association for Computational Linguistics (2006)Google Scholar
  39. 39.
    Ward, W.: Understanding spontaneous speech: the Phoenix system. In: Proceedings of the 1991 International Conference on Acoustics, Speech, and Signal Processing, ICASSP 1991, pp. 365–367. IEEE Computer Society, Washington, DC (1991)CrossRefGoogle Scholar
  40. 40.
    Yasavur, U., Amini, R., Lisetti, C., Rishe, N.: Ontology-based named entity recognizer for behavioral health. In: The Twenty-Sixth International FLAIRS Conference (2013)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.School of Computing and Information SciencesFlorida International UniversityMiamiUSA

Personalised recommendations