Fully Automated Generation of Question-Answer Pairs for Scripted Virtual Instruction

  • Pascal Kuyten
  • Timothy Bickmore
  • Svetlana Stoyanchev
  • Paul Piwek
  • Helmut Prendinger
  • Mitsuru Ishizuka
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7502)


We introduce a novel approach for automatically generating a virtual instructor from textual input only. Our fully implemented system first analyzes the rhetorical structure of the input text and then creates various question-answer pairs using patterns. These patterns have been derived from correlations found between rhetorical structure of monologue texts and question-answer pairs in the corresponding dialogues. A selection of the candidate pairs is verbalized into a diverse collection of question-answer pairs. Finally the system compiles the collection of question-answer pairs into scripts for a virtual instructor. Our end-to-end system presents questions in pre-fixed order and the agent answers them. Our system was evaluated with a group of twenty-four subjects. The evaluation was conducted using three informed consent documents of clinical trials from the domain of colon cancer. Each of the documents was explained by a virtual instructor using 1) text, 2) text and agent monologue, and 3) text and agent performing question-answering. Results show that an agent explaining an informed consent document did not provide significantly better comprehension scores, but did score higher on satisfaction, compared to two control conditions.


Dialogue Generation Rhetorical Structure Theory Medical Documents 


  1. 1.
    Piwek, P., Hernault, H., Prendinger, H., Ishizuka, M.: T2D: Generating Dialogues Between Virtual Agents Automatically from Text. In: Pelachaud, C., Martin, J.-C., André, E., Chollet, G., Karpouzis, K., Pelé, D. (eds.) IVA 2007. LNCS (LNAI), vol. 4722, pp. 161–174. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  2. 2.
    André, E., Rist, T., van Mulken, S., Klesen, M., Baldes, S.: The Automated Design of Believable Dialogues for Animated Presentation Teams. In: Embodied Conversational Agents. MIT Press, Cambridge (2000)Google Scholar
  3. 3.
    Van Deemter, K., Krenn, B., Piwek, P., Klesen, M., Schroeder, M., Baumann, S.: Fully Generated Scripted Dialogue for Embodied Agents. Artificial Intelligence Journal 172(10), 1219–1244 (2008)MATHCrossRefGoogle Scholar
  4. 4.
    Stoyanchev, S., Piwek, P., Prendinger, H.: Comparing Modes of Information Presentation: Text versus ECA and Single versus Two ECAs. In: Vilhjálmsson, H.H., Kopp, S., Marsella, S., Thórisson, K.R. (eds.) IVA 2011. LNCS, vol. 6895, pp. 377–383. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  5. 5.
    Mannem, P., Prasad, R.P., Joshi, A.: Question Generation from Paragraphs at UPenn. In: Proc. of QG 2010: The Third Workshop on Question Generation, Pitsburg, PA, pp. 84–91 (2010)Google Scholar
  6. 6.
    Hernault, H., Prendinger, H.: duVerle, D. and Ishizuka, M. HILDA: A Discourse Parser Using Support Vector Machine Classification. Dialogue and Discourse 1(3), 1–33 (2010)Google Scholar
  7. 7.
    Chen, W., Mostow, J., Aist, G.: Using Automatic Question Generation to Evaluate Questions Generated by Children. In: Question Generation: Papers from the 2011 AAAI Fall Symposium (2011)Google Scholar
  8. 8.
    Gates, D., Aist, G., Mostow, J., McKeown, M., Bey, J.: How to Generate Cloze Questions from Definitions: A Syntactic Approach. In: Question Generation: Papers from the 2011 AAAI Fall Symposium (2011)Google Scholar
  9. 9.
    Nouri, E., Artstein, R., Leuski, A., Traum, D.: Augmenting Conversational Characters with Generated Question-Answer Pairs. In: Question Generation: Papers from the 2011 AAAI Fall Symposium (2011)Google Scholar
  10. 10.
    Heilman, M.: Automatic Factual Question Generation from Text. Ph.D. Dissertation, Carnegie Mellon University, CMU-LTI-11-004CMU-LTI-09-013 (2011)Google Scholar
  11. 11.
    Mann, W.C., Thompson, S.A.: Rhetorical structure theory: Toward a functional theory of text organization. Text 8(3), 243–281 (1988)Google Scholar
  12. 12.
    Piwek, P., Stoyanchev, S.: Data-oriented monologue-to-dialogue generation. In: Proc. Of 14th Annual Meeting of ACL, Short Papers, Portland, Oregon, pp. 242–247 (2011)Google Scholar
  13. 13.
    Bickmore, T., Picard, R.: Establishing and Maintaining Long-Term Human-Computer Relationships. ACM Transactions on Computer Human Interaction 12, 293–327 (2005)CrossRefGoogle Scholar
  14. 14.
    Cassell, J., Vilhjálmsson, H., Bickmore, T.: BEAT. In: Conference BEAT: The Behavior Expression Animation Toolkit, pp. 477–486 (2001)Google Scholar
  15. 15.
    Kuyten, P., Hernault, H., Prendinger, H., Ishizuka, M.: Evaluating HILDA in the CODA Project: A Case Study in Question Generation Using Automatic Discourse Analysis. In: Question Generation: Papers from the 2011 AAAI Fall Symposium (2011)Google Scholar
  16. 16.
    Wogalter, M.S., Howe, J.E., Sifuentes, A.H., Luginbuhl, J.: On the adequacy of legal documents: factors that influence informed consent. Ergonomics 42, 593–613 (1999)CrossRefGoogle Scholar
  17. 17.
    Sugarman, J., Lavori, P.W., Boeger, M., Cain, C., Edson, R., Morrison, V., Yeh, S.S.: Evaluating the quality of informed consent. Clinical Trials 2, 34 (2005)CrossRefGoogle Scholar
  18. 18.
    Sudore, R., Landefeld, C., Williams, B., Barnes, D., Lindquist, K., Schillinger, D.: *Use of a modified informed consent process among vulnerable patients: a descriptive study. J. Gen. Intern. Med. 21, 867–873 (2006)CrossRefGoogle Scholar
  19. 19.
    Bickmore, T., Pfeifer, L., Paasche-Orlow, M.: Using Computer Agents to Explain Medical Documents to Patients with Low Health Literacy. Patient Education and Counseling 75, 315–320 (2009)CrossRefGoogle Scholar
  20. 20.
    Fernando, R.: Automated Explanation of Research Informed Consent by Embodied Conversational Agents. College of Computer and Information Science. MS. Northeastern University, Boston (2009)Google Scholar
  21. 21.
  22. 22.
    Cassell, J., Vilhjálmsson, H., Bickmore, T.: BEAT: The Behavior Expression Animation Toolkit. In: Conference BEAT: The Behavior Expression Animation Toolkit, pp. 477–486 (2001)Google Scholar
  23. 23.
  24. 24.
    Yao, X., Tosch, E., Chen, G., Nouri, E., Artstein, R., Leuski, A., Sagae, K., Traum, D.: Creating conversational characters using question generation tools. Dialogue and Discourse 3(2), 125–146 (2012)Google Scholar
  25. 25.
    Georg, G., Cavazza, M., Pelachaud, C.: Visualizing the Importance of Medical Recommendations with Conversational Agents. In: Prendinger, H., Lester, J.C., Ishizuka, M. (eds.) IVA 2008. LNCS (LNAI), vol. 5208, pp. 380–393. Springer, Heidelberg (2008), doi:10.1007/978-3-540-85483-8_39CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Pascal Kuyten
    • 1
  • Timothy Bickmore
    • 2
  • Svetlana Stoyanchev
    • 3
  • Paul Piwek
    • 4
  • Helmut Prendinger
    • 5
  • Mitsuru Ishizuka
    • 1
  1. 1.Graduate School of Information Science & TechnologyThe University of TokyoJapan
  2. 2.College of Computer and Information ScienceNortheastern UniversityBostonUSA
  3. 3.Spoken Language Processing Group, Department of Computer ScienceColumbia UniversityNew YorkUSA
  4. 4.NLG Group, Centre for Research in ComputingThe Open UniversityWalton HallUK
  5. 5.National Institute of InformaticsTokyoJapan

Personalised recommendations