Journal on Multimodal User Interfaces

, Volume 8, Issue 4, pp 381–398 | Cite as

Let’s talk! speaking virtual counselor offers you a brief intervention

  • Ugan Yasavur
  • Christine Lisetti
  • Naphtali Rishe
Original Paper


We developed a virtual counseling system which can deliver brief alcohol health interventions via a 3D anthropomorphic speech-enabled interface—a new field for spoken dialog interactions with intelligent virtual agents in the health domain. We present our spoken dialog system design and its evaluation. We developed our dialog system based on Markov decision processes framework and optimized it by using reinforcement learning algorithms with data we collected from real user interactions. The system begins to learn optimal dialog strategies for initiative selection and for the type of confirmations that it uses during the interaction. We compared the unoptimized system with the optimized system in terms of objective measures (e.g. task completion) and subjective measures (e.g. ease of use, future intention to use the system) and obtained positive results.


Spoken dialog systems Reinforcement learning Intelligent virtual agents and avatars Conversational agents  Alcohol Healthy lifestyle screening Behavior change brief intervention 



Part of this research was funded by grants from the National Science Foundation HRD-0833093, IIP-1338922, IIP- 1237818.


  1. 1.
    Babor TF, Grant M (1992) Programme on substance abuse: project on identification and management of alcohol-related problems. Report on phase II, an randomized clinical trial of brief interventions in primary health careGoogle Scholar
  2. 2.
    Bickmore T, Gruber A (2010) Relational agents in clinical psychiatry. Harv Rev Psychiatry 18(2):119–130CrossRefGoogle Scholar
  3. 3.
    Bickmore TW, Puskar K, Pfeifer LM, Sereika SM (2010) Maintaining reality: relational agents for antipsychotic medication adherence. Interact Comput 22(4):276–288CrossRefGoogle Scholar
  4. 4.
    Bickmore TW, Schulman D, Sidner CL (2011) A reusable framework for health counseling dialogue systems based on a behavioral medicine ontology. J Biomed Inf 44(2):183–197CrossRefGoogle Scholar
  5. 5.
    Bickmore TW (2005) Establishing and maintaining long-term human-computer relationships. ACM Trans Comput Hum 12(2):293–327CrossRefGoogle Scholar
  6. 6.
    Bohus D, Rudnicky AI (2009) The ravenclaw dialog management framework: architecture and systems. Comput Speech Lang 23(3):332–361CrossRefGoogle Scholar
  7. 7.
    Bouchery EE, Harwood HJ, Sacks JJ, Simon CJ, Brewer RD (2011) Economic costs of excessive alcohol consumption in the US, 2006. Am J Prev Med 41(5):516–524CrossRefGoogle Scholar
  8. 8.
    Chandramohan S, Geist M, Pietquin O (2010) Sparse approximate dynamic programming for dialog management. In: Proceedings of the 11th annual meeting of the special interest group on discourse and dialogue, SIGDIAL’10. Association for Computational Linguistics, Stroudsburg, pp 107–115Google Scholar
  9. 9.
    Chi M, Van Lehn K, Litman D (2010) Do micro-level tutorial decisions matter: applying reinforcement learning to induce pedagogical tutorial tactics. In: Intelligent tutoring systems. Springer, Berlin, pp 224–234Google Scholar
  10. 10.
    de Rosis F, Novielli N, Carofiglio V, Cavalluzzi A, De Carolis B (2006) User modeling and adaptation in health promotion dialogs with an animated characterGoogle Scholar
  11. 11.
    Ferguson G, Allen JF et al (1998) Trips: an integrated intelligent problem-solving assistant. In: Proceedings of the national conference on artificial intelligence. Wiley, New York pp 567–573Google Scholar
  12. 12.
    Ferguson G, Quinn J, Horwitz C, Swift M, Allen J, Galescu L (2010) Towards a personal health management assistant. J Biomed Inf 43(5 Suppl):S13–S16CrossRefGoogle Scholar
  13. 13.
    Frampton M, Lemon O (2009) Recent research advances in reinforcement learning in spoken dialogue systems. Knowl Eng Rev 24(4):375–408CrossRefGoogle Scholar
  14. 14.
    Georgila K, Henderson J, Lemon O (2006) User simulation for spoken dialogue systems, learning and evaluation. In: In Interspeech/ICSLPGoogle Scholar
  15. 15.
    Georgila K, Wolters MK, Moore JD (2010) Learning dialogue strategies from older and younger simulated users. In: Proceedings of the 11th annual meeting of the special interest group on discourse and dialogue, SIGDIAL’10. Association for Computational Linguistics, Stroudsburg, pp 103–106Google Scholar
  16. 16.
    Hone KS, Graham R (2000) Towards a tool for the subjective assessment of speech system interfaces (SASSI). Nat Lang Eng 6(3,4):287–303CrossRefGoogle Scholar
  17. 17.
    Henderson J, Lemon O, Georgila K (2008) Hybrid reinforcement/supervised learning of dialogue policies from fixed data sets. Comput Linguist 34(4):487–511CrossRefGoogle Scholar
  18. 18.
    Hester RK, Squires DD, Delaney HD (2005) The drinker’s check-up: 12-month outcomes of a controlled clinical trial of a stand-alone software program for problem drinkers. J Subst Abuse Treat 28(2):159–169CrossRefGoogle Scholar
  19. 19.
    Humeniuk R, Henry-Edwards S, Ali R, Poznyak V, Monteiro MG, World Health Organization et al (2010) The assist-linked brief intervention for hazardous and harmful substance use: a manual for use in primary care/prepared by R. Humeniuk et al. 2010Google Scholar
  20. 20.
    Jurčíček F, Thomson B, Young S (2012) Reinforcement learning for parameter estimation in statistical spoken dialogue systems. Comput Speech Lang 26(3):168–192CrossRefGoogle Scholar
  21. 21.
    Kennedy CM, Powell J, Payne TH, Ainsworth J, Boyd A, Buchan I (2012) Active assistance technology for health-related behavior change: an interdisciplinary review. J Med Internet Res 14(3)Google Scholar
  22. 22.
    Lemon O, Pietquin O et al (2007) Machine learning for spoken dialogue systems. In: Proceedings of the European conference on speech communication and technologies (Interspeech’07), pp 2685–2688Google Scholar
  23. 23.
    Levin E, Levin A (2006) Evaluation of spoken dialogue technology for real-time health data collection. J Med Internet Res 8(4):e30CrossRefGoogle Scholar
  24. 24.
    Levin E, Pieraccini R, Eckert W (1998) Using markov decision process for learning dialogue strategies. In: Proceedings of the 1998 IEEE international conference on acoustics, speech and signal processing, vol 1. IEEE Press, New York, pp 201–204Google Scholar
  25. 25.
    Li W, Glass J, Roy N, Teller S (2013) Probabilistic dialogue modeling for speech-enabled assistive technology. In: Proceedings of the fourth workshop on speech and language processing for assistive technologies. Association for Computational Linguistics, Grenoble, pp 67–72Google Scholar
  26. 26.
    Lisetti C, Amini R, Yasavur U, Rishe N (2013) I can help you change! an empathic virtual agent delivers behavior change health interventions. ACM TMIS 4(4):19Google Scholar
  27. 27.
    Miller WR, Wilbourne PL (2002) Mesa grande: a methodological analysis of clinical trials of treatments for alcohol use disorders. Addiction 97(3):265–277CrossRefGoogle Scholar
  28. 28.
    Misu T, Georgila K, Leuski A, Traum D (2012) Reinforcement learning of question-answering dialogue policies for virtual museum guides. In: Proceedings of the 13th annual meeting of the special interest group on discourse and dialogue. Association for Computational Linguistics, pp 84–93Google Scholar
  29. 29.
    Mokdad AH, Marks JS, Stroup DF, Gerberding JL (2004) Actual causes of death in the united states, 2000. JAMA 291(10):1238–1245CrossRefGoogle Scholar
  30. 30.
    Morbini F, DeVault D, Sagae K, Gerten J, Nazarian A, Traum D (2012) FLoReS: a forward looking, reward seeking, dialogue manager. In: 4th international workshop on spoken dialog systems, ParisGoogle Scholar
  31. 31.
    Morbini F, Forbell E, DeVault D, Sagae K, Traum DR, Rizzo AA (2012) A mixed-initiative conversational dialogue system for healthcare. In: Proceedings of the 13th annual meeting of the special interest group on discourse and dialogue, SIGDIAL’12. Association for Computational Linguistics, Stroudsburg, pp 137–139Google Scholar
  32. 32.
    Moyer A, Finney JW, Swearingen CE, Vergun P (2002) Brief interventions for alcohol problems: a meta-analytic review of controlled investigations in treatment-seeking and non-treatment-seeking populations. Addiction 97(3):279–292CrossRefGoogle Scholar
  33. 33.
    NIAAA (2006) Niaaa alcohol alert no. 66: Brief interventionsGoogle Scholar
  34. 34.
    NIAAA (2007) Helping patients who drink too much: a clinician’s guideGoogle Scholar
  35. 35.
    NIAAA (2007) What colleges need to know now: an update on college drinking researchGoogle Scholar
  36. 36.
    Paek T, Pieraccini R (2008) Automating spoken dialogue management design using machine learning: an industry perspective. Speech Commun 50(8):716–729CrossRefGoogle Scholar
  37. 37.
    Papangelis A, Kouroupas N, Karkaletsis V, Makedon F (2012) An adaptive dialogue system with online dialogue policy learning. In: Proceedings of the 7th hellenic conference on artificial Intelligence: theories and applications, SETN’12. Springer, Berlin, pp 323–330Google Scholar
  38. 38.
    Portnoy DB, Scott-Sheldon LJ, Johnson BT, Carey MP (2008) Computer-delivered interventions for health promotion and behavioral risk reduction: a meta-analysis of 75 randomized controlled trials, 1988–2007. Prev Med 47(1):3–16CrossRefGoogle Scholar
  39. 39.
    Riper H, Spek V, Boon B, Conijn B, Kramer J, Martin-Abello K, Smit F (2011) Effectiveness of e-self-help interventions for curbing adult problem drinking: a meta-analysis. J Med Internet Res 13(2):e42CrossRefGoogle Scholar
  40. 40.
    Saitz R, Palfai TP, Freedner N, Winter MR, Macdonald A, Lu J, Ozonoff AL, Rosenbloom DL, Dejong W (2007) Screening and brief intervention online for college students: the ihealth study. Alcohol Alcohol 42(1):28–36CrossRefGoogle Scholar
  41. 41.
    Schatzmann J, Georgila K, Young S (2005) Quantitative evaluation of user simulation techniques for spoken dialogue systems. In: 6th SIGdial workshop on DISCOURSE and DIALOGUEGoogle Scholar
  42. 42.
    Schulman D, Bickmore TW, Sidner CL (2011) An intelligent conversational agent for promoting long-term health behavior change using motivational interviewing. In: Association for the advancement of artificial intelligence (AAAI) spring symposium series. Association for the advancement of artificial intelligence, pp 61–64.
  43. 43.
    Singh S, Kearns M, Litman DJ, Walker MA et al (2000) Empirical evaluation of a reinforcement learning spoken dialogue system. In: AAAI/IAAI, pp 645–651 Google Scholar
  44. 44.
    Singh S, Litman D, Kearns M, Walker M (2002) Optimizing dialogue management with reinforcement learning: experiments with the NJFun system. J Artif Intell Res 16:105–133Google Scholar
  45. 45.
    Steinberg KL, Roffman RA, Carroll KM, McRee B, Babor TF, Miller M, Kadden R, Duresky D, Stephens R (2005) Brief counseling for marijuana dependence: a manual for treating adults. Center for substance abuse treatment, substance abuse and mental health services administration, RockvilleGoogle Scholar
  46. 46.
    Sutton RS, Barto AG (1998) Reinforcement learning: an introduction, vol 1. Cambridge University Press, CambridgeGoogle Scholar
  47. 47.
    Sutton S, Cole R, De Villiers J, Schalkwyk J, Vermeulen P, Macon M, Yan Y, Kaiser E, Rundle B, Shobaki K et al (1998) Universal speech tools: the CSLU toolkit. In: Proceedings of the international conference on spoken language processing (ICSLP). Sydney, pp 3221–3224Google Scholar
  48. 48.
    Traum D, Larsson S (2003) The Information state approach to dialogue management. Kluwer Academic, New York, pp 325–353Google Scholar
  49. 49.
    Turunen M, Hakulinen J, Ståhl O, Gambäck B, Hansen P, de la Cámara R, Smith C, Charlton D, Cavazza M (2011) Multimodal and mobile conversational health and fitness companions. Comput Speech Lang 25(2):192–209CrossRefGoogle Scholar
  50. 50.
    Walters ST, Hester RK, Chiauzzi E, Miller E (2005) Demon rum: high-tech solutions to an age-old problem. Alcohol Clin Exp Res 29(2):270–277CrossRefGoogle Scholar
  51. 51.
    Walters ST, Neighbors C (2005) Feedback interventions for college alcohol misuse: what, why and for whom? Addict Behav 30(6):1168–1182CrossRefGoogle Scholar
  52. 52.
    Ward W (1991) Understanding spontaneous speech: the phoenix system. In: Proceedings of the acoustics, speech, and signal processing. ICASSP-91, 1991 international conference, ICASSP’91. IEEE Computer Society, Washington, DC, pp 365–367Google Scholar
  53. 53.
    White A, Kavanagh D, Stallman H, Klein B, Kay-Lambkin F, Proudfoot J, Drennan J, Connor J, Baker A, Hines E, Young R (2010) Online alcohol interventions: a systematic review. J Med Internet Res 12(5):e62CrossRefGoogle Scholar
  54. 54.
    Williams JD (2008) The best of both worlds: unifying conventional dialog systems and pomdps. In: INTERSPEECH, pp 1173–1176Google Scholar
  55. 55.
    Williams JD, Young S (2007) Partially observable markov decision processes for spoken dialog systems. Comput Speech Lang 21(2):393–422CrossRefGoogle Scholar
  56. 56.
    Young S, Gasic M, Thomson B, Williams JD (2013) Pomdp-based statistical spoken dialog systems: a review. Proc. EEE 101(5):1160–9219, doi: 10.1109/JPROC.2012.2225812
  57. 57.
    Young S, Gašić M, Keizer S, Mairesse F, Schatzmann J, Thomson B, Yu K (2010) The hidden information state model: a practical framework for pomdp-based spoken dialogue management. Comput Speech Lang 24(2):150–174CrossRefGoogle Scholar
  58. 58.
    Young S, Gašić M, Keizer S, Mairesse F, Schatzmann J, Thomson B, Yu K (2010) The hidden information state model: a practical framework for pomdp-based spoken dialogue management. Comput Speech Lang 24(2):150–174CrossRefGoogle Scholar

Copyright information

© OpenInterface Association 2014

Authors and Affiliations

  • Ugan Yasavur
    • 1
  • Christine Lisetti
    • 1
  • Naphtali Rishe
    • 1
  1. 1.School of Computing and Information SciencesFlorida International UniversityMiamiUSA

Personalised recommendations