Abstract
We report an experiment to evaluate DQGen’s performance in generating three types of distractors for diagnostic multiple-choice cloze (fill-in-the-blank) questions to assess children’s reading comprehension processes. Ungrammatical distractors test syntax, nonsensical distractors test semantics, and locally plausible distractors test inter-sentential processing. 27 knowledgeable humans rated candidate answers as correct, plausible, nonsensical, or ungrammatical without knowing their intended type or whether they were generated by DQGen, written by other humans, or correct. Surprisingly, DQGen did significantly better than humans at generating ungrammatical distractors and slightly better than them at generating nonsensical distractors, albeit worse at generating plausible distractors. Vetting its output and writing distractors only when necessary would take half as long as writing them all, and improve their quality.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Mostow, J., Beck, J.E., Bey, J., Cuneo, A., Sison, J., Tobin, B., Valeri, J.: Using Automated Questions to Assess Reading Comprehension, Vocabulary, and Effects of Tutorial Interventions. Technology, Instruction, Cognition and Learning 2(1–2), 97–134 (2004)
Mostow, J., Chen, W.: Generating instruction automatically for the reading strategy of self-questioning. In: Proceedings of the 14th International Conference on Artificial Intelligence in Education, pp. 465–472. IOS Press, Brighton, UK (2009)
Gates, D., Aist, G., Mostow, J., Mckeown, M., Bey, J.: How to generate cloze questions from definitions: a syntactic approach. In: Proceedings of the AAAI Symposium on Question Generation 2011. AAAI Press, Arlington, VA
Chen, W., Mostow, J., Aist, G.S.: Recognizing Young Readers’ Spoken Questions. International Journal of Artificial Intelligence in Education 21(4), 255–269 (2013)
Mitkov, R., Ha, L.A., Karamanis, N.: A Computer-aided Environment for Generating Multiple Choice Test Items. Natural Language Engineering 12(2), 177–194 (2006)
Agarwal, M., Mannem, P.: Automatic gap-fill question generation from text books. In: Proceedings of the 6th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 56–64. Association for Computational Linguistics (2011)
Sumita, E., Sugaya, F., Yamamoto, S.: Measuring non-native speakers’ proficiency of English by using a test with automatically-generated fill-in-the-blank questions. In: Proceedings of the Second Workshop on Building Educational Applications Using NLP, pp. 61–68. Association for Computational Linguistics, Ann Arbor, MI (2005)
Lee, J., Seneff, S.: Automatic generation of cloze items for prepositions. In: INTERSPEECH, pp. 2173–2176 (2007)
Lin, Y.-C., Sung, L.-C., Chen, M.C.: An automatic multiple-choice question generation scheme for english adjective understanding. In: Workshop on Modeling, Management and Generation of Problems/Questions in eLearning, the 15th International Conference on Computers in Education (ICCE 2007), pp. 137–142 (2007)
Huang, Y.-T., Chen, M.C., Sun, Y.S.: Personalized automatic quiz generation based on proficiency level estimation. In: 20th International Conference on Computers in Education (ICCE 2012), Singapore (2012)
Ming, L., Calvo, R.A., Aditomo, A., Pizzato, L.A.: Using Wikipedia and Conceptual Graph Structures to Generate Questions for Academic Writing Support. IEEE Transactions on Learning Technologies 5(3), 251–263 (2012)
Graesser, A.C., Bertus, E.L.: The Construction of Causal Inferences While Reading Expository Texts on Science and Technology. Scientific Studies of Reading 2(3), 247–269 (1998)
van den Broek, P., Everson, M., Virtue, S., Sung, Y., Tzeng, Y.: Comprehension and memory of science texts: Inferential processes and the construction of a mental representation. In: Otero, J.L.J., Graesser, A.C. (eds.) The Psychology of Science Text Comprehension. Erlbaum, Mahwah, NJ (2002)
Kintsch, W.: An Overview of Top-Down and Bottom-Up Effects in Comprehension: The CI Perspective. Discourse Processes A Multidisciplinary Journal 39(2&3), 125–128 (2005)
Mostow, J., Jang, H.: Generating diagnostic multiple choice comprehension cloze questions. NAACL-HLT 2012 7th Workshop on Innovative Use of NLP for Building Educational Applications, pp. 136–146. Association for Computational Linguistics, Montréal (2012)
Mostow, J.: Lessons from project LISTEN: what have we learned from a reading tutor that listens? In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS, vol. 7926, pp. 557–558. Springer, Heidelberg (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this paper
Cite this paper
Huang, YT., Mostow, J. (2015). Evaluating Human and Automated Generation of Distractors for Diagnostic Multiple-Choice Cloze Questions to Assess Children’s Reading Comprehension. In: Conati, C., Heffernan, N., Mitrovic, A., Verdejo, M. (eds) Artificial Intelligence in Education. AIED 2015. Lecture Notes in Computer Science(), vol 9112. Springer, Cham. https://doi.org/10.1007/978-3-319-19773-9_16
Download citation
DOI: https://doi.org/10.1007/978-3-319-19773-9_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-19772-2
Online ISBN: 978-3-319-19773-9
eBook Packages: Computer ScienceComputer Science (R0)