Soft Computing

, Volume 18, Issue 7, pp 1269–1281 | Cite as

Automatic generation of multiple choice questions using dependency-based semantic relations

Methodologies and Application
First Online:

Abstract

In this paper, we present an unsupervised dependency-based approach to extract semantic relations to be applied in the context of automatic generation of multiple choice questions (MCQs). MCQs also known as multiple choice tests provide a popular solution for large-scale assessments as they make it much easier for test-takers to take tests and for examiners to interpret their results. Manual generation of MCQs is a very expensive and time-consuming task and yet they often need to be produced on a large scale and within short iterative cycles. We approach the problem of automated MCQ generation with the help of unsupervised relation extraction, a technique used in a number of related natural language processing problems. The goal of Unsupervised relation extraction is to identify the most important named entities and terminology in a document and then recognise semantic relations between them, without any prior knowledge as to the semantic types of the relations or their specific linguistic realisation. We use these techniques to process instructional texts and identify those facts (terminology, entities, and semantic relations between them) that are likely to be important for assessing test-takers’ familiarity with the instructional material. We investigate an approach to learn semantic relations between named entities by employing a dependency tree model. Our findings show that an optimised configuration of our MCQ generation system is capable of attaining high precision rates, which are much more important than recall in the automatic generation of MCQs. We also carried out a user-centric evaluation of the system, where subject domain experts evaluated automatically generated MCQ items in terms of readability, usefulness of semantic relations, relevance, acceptability of questions and distractors and overall MCQ usability. The results of this evaluation make it possible for us to draw conclusions about the utility of the approach in practical e-Learning applications.

Keywords

E-Learning Automatic assessment Natural language processing Information extraction Dependency tree Unsupervised relation extraction Multiple choice questions generation Biomedical domain 
This is a preview of subscription content, log in to check access.

References

  1. Afzal N, Mitkov R, Farzindar A (2011) Unsupervised Relation extraction using dependency trees for automatic generation of multiple-choice questions. In: Butz C, Lingras P (eds) Proceedings of the Canadian AI 2011, LNAI 6657. Springer, Heidelberg, pp 32–43Google Scholar
  2. Agichtein E, Gravano L (2000) Snowball: Extracting Relations from Large Plaintext Collections. In: Proceedings of the 5th ACM international conference on digital librariesGoogle Scholar
  3. Bikel DM, Miller S, Schwartz R, Weischedel R (1998) Nymble: a high-performance learning name-finder. In Proceedings of the conference on applied natural language processingGoogle Scholar
  4. Brown J, Frishkoff G, Eskenazi M (2005) Automatic question generation for vocabulary assessment. In: Proceeding of HLT/EMNLP. Vancouver, BCGoogle Scholar
  5. Caraballo SA (1999) Automatic construction of a hypernym-labeled noun hierarchy from text. In: Proceedings of 37th annual meeting of the association for computational linguistics, pp 120–126Google Scholar
  6. Carlsson C, Brunelli M, Mezei J (2012) Decision making with a fuzzy ontology. Soft Comput 16(7):1143–1152CrossRefGoogle Scholar
  7. Chen C-Y, Liou H-C, Chang JS (2006) FAST—an automatic generation system for grammar tests. In: Proceedings of COLING/ACL interactive presentation sessions, SydneyGoogle Scholar
  8. Chen W, Aist G, Mostow J (2009) Generating questions automatically from informational text. In: Proceedings of the 2nd workshop on question generation. BrightonGoogle Scholar
  9. Cohen AM, Hersh WR (2005) A survey of current work in biomedical text mining. Brief Bioinform 6(1):57–71Google Scholar
  10. Cohen J (1968) Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol BullGoogle Scholar
  11. Corney DP, Jones D, Buxton B, Langdon W (2004) BioRAT: extracting biological information from full-length papers. Bioinformatics 20:3206–3213CrossRefGoogle Scholar
  12. Cover T, Thomas J (1991) Elements of information theory. Wiley, New YorkCrossRefMATHGoogle Scholar
  13. Dagan I, Lee L, Pereira F (1997) Similarity-based methods for word sense disambiguation. In: Proceedings of the 35th annual meeting of the association for computational linguistics, Madrid, p 56.63Google Scholar
  14. Dagan I, Lee L, Pereira F (1999) Similarity-based models of word cooccurrence probabilities. Mach Learn J 34(1–3):43–69Google Scholar
  15. Das R, Elikkottil A (2010) Auto-summarizer to aid a Q/A system. Int J Comput Appl 1(1):113–117Google Scholar
  16. De Maio C, Fenza G, Loia V, Senatore S (2009) Towards an automatic fuzzy ontology generation. In: Proceedings of IEEE international conference on fuzzy systems, pp 1044–1049Google Scholar
  17. Dhillon IS, Mallela S, Kumar R (2002) Enhanced word clustering for hierarchical text classification (Tech. Rep. Nos. TR-02-17). Austin: Department of Computer Sciences, University of TexasGoogle Scholar
  18. Farzindar A, Lapalme G (2004) LetSum, an automatic Legal Text Summarizing system. In: Gordon Thomas F (ed) Legal Knowledge and Information Systems, Jurix 2004: the 7th annual conference. IOS Press, Berlin, pp 11–18Google Scholar
  19. Firth JR (1957) A synopsis of linguistic theory 1930–1955. Studies in Linguistic Analysis. Blackwell, Oxford, pp 1–32Google Scholar
  20. Gates D (2008) Generating Look-Back Strategy Questions from Expository Texts. In: Workshop on the question generation shared task and evaluation challenge. NSF, ArlingtonGoogle Scholar
  21. Graesser A, Person N (1994) Question asking during tutoring. Am Educ Res J 31:104–137CrossRefGoogle Scholar
  22. Graesser AC, Chipman P, Haynes BC, Olney A (2005) Autotutor: an intelligent tutoring system with mixed-initiative dialogue. IEEE Trans Educ 48(4):612–618CrossRefGoogle Scholar
  23. Grefenstette G (1994) Explorations in automatic Thesaurus discovery, vol. 278 of Kluwer International Series in Engineering and Computer Science. Kluwer, BostonGoogle Scholar
  24. Gronlund N (1982) Constructing achievement tests. Prentice Hall, New YorkGoogle Scholar
  25. Harris Z (1954) Distributional structure. Word 10(23):146–162Google Scholar
  26. Harshman R (1970) Foundations of the parafac procedure: Models and conditions for an “explanatory” multi-modal factor analysis. In: UCLA Working Papers in Phonetics, vol 16Google Scholar
  27. Hasegawa T, Sekine S, Grishman R (2004) Discovering relations among named entities from large corpora. In: Proceedings of ACL’04Google Scholar
  28. Hatzivassiloglou V (1996) Do we need linguistics when we have statistics? A comparative analysis of the contributions of linguistic cues to a statistical word grouping system. In: Judith K, Philip R (eds) The balancing act: combining symbolic and statistical approaches to language, chapter 4. MIT Press, Cambridge, pp 67–94Google Scholar
  29. Hirschman L, Mani I (2003) Evaluation. In: Mitkov R (ed) The Oxford Handbook of Computational Linguistics. Oxford University Press, UK, pp 414–429Google Scholar
  30. Hodges PE, McKee AH, Davis BP, Payne WE, Garrels JI (1999) The Yeast Proteome Database (YPD): a model for the organization and presentation of genomewide functional data. Nucleic Acids Res 27(1): 69–73Google Scholar
  31. Hoshino A, Nakagawa H (2007) Assisting cloze test making with a web application. In: Proceedings of society for information technology and teacher education international conference, ChesapeakeGoogle Scholar
  32. Huang M, Zhu X, Payan GD, Qu K, Li M (2004) Discovering patterns to extract protein-protein interactions from full biomedical texts. Bioinformatics, pp 3604–3612Google Scholar
  33. Kalady S, Elikkottil A, Das R (2010) Natural language question generation using syntax and keywords. In: Proceedings of the 3rd workshop on question generationGoogle Scholar
  34. Karamanis N, Ha LA, Mitkov R (2006) Generating multiple-choice test items from medical text: A pilot study. In: Proceedingd of the 4th international natural language generation conference, (July), pp 111–113Google Scholar
  35. Kullback S, Leibler R (1951) On information and sufficiency. Ann Math Stat 22:79–86CrossRefMATHMathSciNetGoogle Scholar
  36. Lapata M, Keller F, McDonald S (2001) Evaluating smoothing algorithms against plausibility judgements. In: Proceedings of the 39th annual meeting of the association for computational linguistics (ACL-2001), Toulouse, pp 346–353Google Scholar
  37. Lin D (1998) Automatic retrieval and clustering of similar words. In: Proceedings of international conference on computational linguistics and the annual meeting of the association for Computational LinguisticsGoogle Scholar
  38. Lin J (1991) Divergence measures based on the Shannon entropy. IEEE Trans Inform Theory 37(1):145–151CrossRefMATHMathSciNetGoogle Scholar
  39. Martin EP, Bremer E, Guerin G, DeSesa M-C, Jouve O (2004) Analysis of protein/protein interactions through biomedical literature: text mining of abstracts vs. text mining of full text articles. Springer, Berlin, pp 96–108Google Scholar
  40. Mitkov R, An LA (2003) Computer-aided generation of multiple-choice tests. In: Proceedings of the HLT/NAACL 2003 workshop on building educational applications using natural language processing, Edmonton, pp 17–22Google Scholar
  41. Mitkov R, Ha LA, Karamanis N (2006) A computer-aided environment for generating multiple-choice test items. Natural Language Engineering 12(2). Cambridge University Press, Cambridge, pp 177–194Google Scholar
  42. Mostow J, Chen W Generating Instruction Automatically for the Reading Strategy of Self-Questioning. In: Proceedings of the 14th international conference on artificial intelligence in Education, BrightonGoogle Scholar
  43. Nielsen R (2008) Question generation: Proposed challenge tasks and their evaluation. In: Proceedings of the workshop on the question generation shared task and evaluation, challengeGoogle Scholar
  44. Palmer M, Kingsbury P, Gildea D (2005) The proposition bank: an annotated corpus of semantic roles. Comput Linguist 31(1): 71–106Google Scholar
  45. Papasalouros A, Kanaris K, Konstantinos K (2008) Automatic generation of multiple choice questions from domain ontologies. In: Proceeding of IADIS international conference e-learningGoogle Scholar
  46. Paroubek P, Chaudiron S, Hirschman L (2007) Principles of evaluation in natural language processing. TAL 48(1/2007):7–31Google Scholar
  47. Pereira F, Tishby N, Lee L (1993) Distributional clustering of similar words. In: Proceedings of the 31st annual meeting of the association for computational linguistics (ACL-1993), Columbus, pp 183–190Google Scholar
  48. Pradhan S, Hacioglu K, Krugler V, Ward W, Martin JH, Jurafsky D (2005) Support vector learning for semantic argument classification. Mach Learn 60(1):11–39CrossRefGoogle Scholar
  49. Rao CR (1983) Diversity: its measurement, decomposition, apportionment and analysis. Indian J Stat 44(A):1–22Google Scholar
  50. Schwartz L, Aikawa T, Pahud M (2004) Dynamic language learning tools. In: Proceedings of the of the 2004 In-STIL/ICALL SymposiumGoogle Scholar
  51. Stevenson M, Greenwood M (2005) A semantic approach to IE pattern induction. In: Proceedings of ACL’05, pp 379–386Google Scholar
  52. Stevenson M, Greenwood M (2009) Dependency pattern models for information extraction. Res Lang ComputGoogle Scholar
  53. Sumita E, Sugaya F, Yamamoto S (2005) Measuring non-native speakers’ proficiency of English using a test with automatically-generated fill-in-the-blank questions. In: Proceedings of the 2nd workshop on building educational applications using NLP, pp 61–68Google Scholar
  54. Tateno J, Sano H, Aizawa H, Nakamura T, Morita Y (2005) Producing english Educational materials form the BNC and releasing them on the Web, IEICE Technical report, TL2005-1826, Tokyo, pp 7–12Google Scholar
  55. Ureel L, Forbus K, Riesbeck C, Birnbaum L (2005) Question generation for learning by reading. In: Proceedings of the AAAI workshop on textual question answering, PittsburghGoogle Scholar
  56. Vanderwende L (2007) Answering and questioning for machine reading. In: Proceedings of the 2007 AAAI spring symposium on machine reading, Stanford Google Scholar
  57. Vanderwende L (2008) The importance of being important: question generation. In: Proceedings of the workshop on the question generation shared task and evaluation challenge, ArlingtonGoogle Scholar
  58. Walker MA, Rambow O, Rogati M (2001) Spot: a trainable sentence planner. In: Proceedings of NAACLGoogle Scholar
  59. Weeds J (2003) Measures and applications of lexical distributional similarity. Ph.D. thesis, University of SussexGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  1. 1.Faculty of Computing and Information Technology (FCIT)King Abdulaziz University, North Branch JeddahJeddahSaudi Arabia
  2. 2.Research Institute for Information and Language Processing (RIILP)University of WolverhamptonWolverhamptonUK

Personalised recommendations