Health Document Explanation by Virtual Agents

  • Timothy W. Bickmore
  • Laura M. Pfeifer
  • Michael K. Paasche-Orlow
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4722)


We describe the design and evaluation of a virtual agent that explains health documents to patients. The prevalence and impact of low health literacy is presented as a motivation for such agents, given that face-to-face interaction with health providers is cited as one of the most effective means of communicating with these patients. We analyze the form and distribution of pointing gestures used by experts in explaining health documents, and use this data to develop a computational model of agent-based document explanation. This model is evaluated in a randomized controlled trial. Preliminary results indicate that patients with low health literacy are more satisfied with health document explanation by a virtual agent compared to a human.


Virtual agent relational agent embodied conversational agent health literacy hand gesture 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ad Hoc Committee on Health Literacy for the Council on Scientific Affairs, A.M.A.: Health literacy: report of the Council on Scientific Affairs. JAMA 281(6) 552–557 (1999) Google Scholar
  2. 2.
    Allen, J., Core, M.: Draft of DMSL: Dialogue Act Markup in Several Layers (1997) Google Scholar
  3. 3.
    Andre, E., Rist, T., Muller, J.: Integrating reactive and scripted behaviors in a life-like presentation agent. In: Proceedings of AGENTS 1998, pp. 261–268 (1998)Google Scholar
  4. 4.
    Baker, D., Parker, R., Williams, M., Clark, S., Nurss, J.: The relationship of patient reading ability to self-reported health and use of services. Am. J. Public Health 87, 1027–1030 (1997)CrossRefGoogle Scholar
  5. 5.
    Beecher, H.K.: Ethics and clinical research. N. Engl. J. Med. 274(24), 1354–1360 (1966)CrossRefGoogle Scholar
  6. 6.
    Bickmore, T., Picard, R.: Establishing and Maintaining Long-Term Human-Computer Relationships. ACM Transactions on Computer Human Interaction 12(2), 293–327 (2005)CrossRefGoogle Scholar
  7. 7.
    Cassell, J., Ananny, M., Basu, A., Bickmore, T., Chong, P., Mellis, D., Ryokai, K., Smith, J., Vilhjálmsson, H., Yan, H.: Shared Reality: Physical Collaboration with a Virtual Peer. In: Proceedings of CHI 2000 (2000)Google Scholar
  8. 8.
    Cassell, J., Stocky, T., Bickmore, T., Gao, Y., Nakano, Y., Ryokai, K., Tversky, D., Vaucelle, C., Vilhjálmsson, H.: MACK: Media lab Autonomous Conversational Kiosk. Imagina 2002 (2002) Google Scholar
  9. 9.
    Cassell, J., Vilhjálmsson, H., Bickmore, T.: BEAT: The Behavior Expression Animation Toolkit. In: SIGGRAPH 2001, pp. 477–486 (2001)Google Scholar
  10. 10.
    Clark, H.H., Brennan, S.E.: Grounding in Communication. In: Resnick, L.B., Levine, J.M., Teasley, S.D. (eds.) Perspectives on Socially Shared Cognition, American Psychological Association, Washington, pp. 127–149 (1991)Google Scholar
  11. 11.
    Clinite, J., Kabat, H.: Improving patient compliance. J. Am. Pharm. Assoc. 16, 74–76 (1976)Google Scholar
  12. 12.
    Davidoff, F.: Time. Ann. Intern. Med. 127, 483–485 (1997)Google Scholar
  13. 13.
    Davis, T.C., Long, S.W., Jackson, R.H., Mayeaux, E.J., George, R.B., Murphy, P.W., Crouch, M.A.: Rapid estimate of adult literacy in medicine: a shortened screening instrument. Fam. Med. 25(6), 391–395 (1993)Google Scholar
  14. 14.
    Doak, C., Doak, L., Root, J.: Teaching patients with low literacy skills, 2nd edn., JB Lippincott, Philadelphia, PA (1996)Google Scholar
  15. 15.
    Frankel, R.: Emotion and the Physician-Patient Relationship. Motivation and Emotion 19(3), 163–173 (1995)CrossRefGoogle Scholar
  16. 16.
    Graesser, A., et al.: AutoTutor: A simulation of a human tutor. Cognitive Systems Research 1 (1999)Google Scholar
  17. 17.
    Grosz, B., Sidner, C.: Attention, Intentions, and the Structure of Discourse. Computational Linguistics 12(3), 175–204 (1986)Google Scholar
  18. 18.
    Kalichman, S., Ramachandran, B., Catz, S.: Adherence to a combination antiretroviral therapies in HIV patients of low health literacy. J. Gen. Intern. Med. 14, 267–273 (1999)CrossRefGoogle Scholar
  19. 19.
    Katz, M.G., Jacobson, T.A., Veledar, E., Kripalani, S.: Patient Literacy and Question-asking Behavior During the Medical Encounter: A Mixed-methods Analysis. J. Gen. Intern. Med. (2007)Google Scholar
  20. 20.
    Kopp, S., Krenn, B., Marsella, S., Marshall, A.N., Pelachaud, C., Pirker, H., Thórisson, K., Vilhjálmsson, H.: Towards a Common Framework for Multimodal Generation: The Behavior Markup Language. In: Gratch, J., Young, M., Aylett, R., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS (LNAI), vol. 4133, Springer, Heidelberg (2006)Google Scholar
  21. 21.
    Lester, J., Towns, S., Callaway, C., Voerman, J., Fitzgerald, P.: Deictic and Emotive Communication in Animated Pedagogical Agents. In: Cassell, J. (ed.) Embodied Conversational Agents, MIT Press, Cambridge (2000)Google Scholar
  22. 22.
    Lester, J., Voerman, J., Towns, S., Callaway, C.: Deictic Believability: Coordinating Gesture, Locomotion, and Speech in Lifelike Pedagogical Agents. Applied Artificial Intelligence 13(4-5), 383–414 (1999)CrossRefGoogle Scholar
  23. 23.
    Madden, E.: Evaluation of outpatient pharmacy patient counseling. J. Am. Pharm. Ass. 13, 437–443 (1973)Google Scholar
  24. 24.
    Moreno, R., Lester, J.C., Mayer, R.E.: Life-Like Pedagogical Agents in Constructivist Multimedia Environments: Cognitive Consequences of their Interaction. In: ED-MEDIA, pp. 741–746 (2000)Google Scholar
  25. 25.
    Morris, L., Halperin, J.: Effects of Written Drug Information on Patient Knowledge and Compliance: A Literature Review. Am. J. Public Health 69(1), 47–52 (1979)Google Scholar
  26. 26.
    Nakano, Y.I., Reinstein, G., Stocky, T., Cassell, J.: Towards a Model of Face-to-Face Grounding. In: Annual Meeting of the Association for Computational Linguistics (2003)Google Scholar
  27. 27.
    Nakatani, C., Traum, D.: Coding discourse structure in dialogue (version 1.0). University of Maryland (1999) Google Scholar
  28. 28.
    Nielsen-Bohlman, L., Panzer, A.M., Hamlin, B., Kindig, D.A.: Institute of Medicine. Health Literacy: A Prescription to End Confusion. In: Committee on Health Literacy, Board on Neuroscience and Behavioral Health, National Academies Press, Washington, DC (2004)Google Scholar
  29. 29.
    Noma, T., Zhao, L., Badler, N.I.: Design of a virtual human presenter. Computer Graphics and Applications, IEEE 20(4), 79–85 (2000)CrossRefGoogle Scholar
  30. 30.
    Paasche-Orlow, M., Greene, S.M., Wagner, E.H.: How health care systems can begin to address the challenge of limited literacy. J. Gen. Intern. Med. 21(8), 884–887 (2006)CrossRefGoogle Scholar
  31. 31.
    Paasche-Orlow, M.K., Parker, R.M., Gazmararian, J.A., Nielsen-Bohlman, L.T., Rudd, R.R.: The prevalence of limited health literacy. J. Gen. Intern. Med. 20(2), 175–184 (2005)CrossRefGoogle Scholar
  32. 32.
    Person, N.K., Graesser, A.C., Bautista, L., Mathews, E.C.: Evaluating Student Learning Gains in Two Versions of AutoTutor. In: Moore, J.D., Redfield, C.L., Johnson, W.L. (eds.) Artificial intelligence in education: AI-ED in the wired and wireless future, pp. 286–293. IOS Press, Amsterdam (2001)Google Scholar
  33. 33.
    Qualls, C., Harris, J., Rogers, W.: Cognitive-Linguistic Aging: Considerations for Home Health Care Environments. In: Rogers, W., Fisk, A. (eds.) Human Factors Interventions for the Health Care of Older Adults, pp. 47–67. Lawrence Erlbaum, Mahwah (2002)Google Scholar
  34. 34.
    Richmond, V., McCroskey, J.: Immediacy. In: Nonverbal Behavior in Interpersonal Relations, pp. 195–217. Allyn & Bacon, Boston (1995)Google Scholar
  35. 35.
    Rickel, J., Johnson, W.L.: Animated Agents for Procedural Traning in Virtual Reality: Perception, Cognition and Motor Control. Applied Artificial Intelligence (1998)Google Scholar
  36. 36.
    Schillinger, D., Piette, J., Grumbach, K., Wang, F., Wilson, C., Daher, C.: Closing the loop: physician communication with diabetic patients who have low health literacy. Arch. Intern. Med. 163(1), 83–90 (2003)CrossRefGoogle Scholar
  37. 37.
    Sugarman, J., Lavori, P.W., Boeger, M., Cain, C., Edson, R., Morrison, V., Yeh, S.S.: Evaluating the quality of informed consent. Clinical Trials 2(1), 34 (2005)CrossRefGoogle Scholar
  38. 38.
    Tannen, D.: Framing in Discourse. Oxford University Press, New York (1993)Google Scholar
  39. 39.
    Weiss, B.: Illiteracy among Medicaid recipients and its relation to health care costs. J. Health Care Poor Underserved 4(5), 99–111 (1994)Google Scholar
  40. 40.
    Weiss, B., Hart, G., McGee, D., D’Estelle, S.: Health status of illerate adults: relation between literacy and health status among persons with low literacy skills. J. Am. Board Fam. Parct. 5, 254–257 (1992)Google Scholar
  41. 41.
    Williams, M., Parker, R., Baker, D., Coates, W., Nurss, J.: The impact of inadequate functional health literacy on patients understanding of diagnosis, prescribed medications, and compliance. Acad. Emerg. Med. 2, 386 (1995)Google Scholar
  42. 42.
    Woods, W.A.: Transition Network Grammars for Natural Language Analysis. In: Grosz, B.J., Jones, K.S., Webber, B.L. (eds.) Readings in Natural Language Processing, pp. 71–88. Morgan Kaufmann Publishers, Los Alamitos (1986)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Timothy W. Bickmore
    • 1
  • Laura M. Pfeifer
    • 1
  • Michael K. Paasche-Orlow
    • 2
  1. 1.Northeastern University College of Computer and Information Science, 360 Huntington Ave WVH202, Boston, MA 02115 
  2. 2.Boston University School of Medicine, 91 East Concord St, Suite 200, Boston, MA 02118 

Personalised recommendations