Skip to main content
SpringerLink
Log in

QDG: A unified model for automatic question-distractor pairs generation

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Generating high-quality complete question sets (for example, the question, answer and distractors) in reading comprehension tasks is challenging and rewarding. This paper proposes a question-distractor joint generation framework (QDG). The framework can automatically generate both questions and distractors given a background text and the specified answer. Our work makes it possible to combine complete multiple-choice reading comprehension questions that can be better applied to educators’ work. While there have been independent studies of question generation and distractor generation in previous studies, there have been few joint question-distractor generation studies. In a past joint generation, distractors could only be constructed by generating questions first and then by sorting the answers with similar words. It was impossible to generate question-distractor pairs in an end-to-end unified joint generation approach. To the best of our knowledge, we are the first to propose an end-to-end question-distractor joint generation framework on the RACE dataset. This paper finds that distractors are somehow relevant to the background articles, by suppressing those related parts, thus enabling the generated questions to be better focused on the relevant parts of the correct answers. The experimental results show that the model achieves a giant breakthrough in the question-distractor pair generation task. The question generation task achieves better performance than baselines. For further evaluation, we also manually people for evaluation to demonstrate the educational relevance of our model in generating high-quality question-distractor pairs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availability

All data included in this study are available upon request by contact with the corresponding author.

References

  1. Adamson D, Bhartiya D, Gujral B, Kedia R, Singh A, Rosé CP (2013) Automatically generating discussion questions. In: AIED. Lecture notes in computer science, vol 7926. pp 81–90, Springer

  2. Cao ND, Aziz W, Titov I (2019) Question answering by reasoning across documents with graph convolutional networks. In: Association for computational linguistics. NAACL-HLT (1) pp 2306–2317

  3. Cao Y, Fang M, Tao D (2019) BAG: Bi-directional attention entity graph convolutional network for multi-hop reasoning question answering. In: NAACL-HLT, (1) pp 357–362, Association for computational linguistics

  4. Chen Y, Wu L, Zaki MJ (2020) Reinforcement learning based graph-to-sequence model for natural question generation. In: ICLR. Open Review net

  5. Cheng Y, Li S, Liu B, Zhao R, Li S, Lin C, Zheng Y (2021) Guiding the growth: Difficulty-controllable question generation through step-by-step rewriting. In: Association for computational linguistics. ACL/IJCNLP (1) pp 5968–5978

  6. Deepak G, Kumar N, Bharadwaj GVSY, Santhanavijayan A (2019) Ontoquest: an ontological strategy for automatic question generation for e-assessment using static and dynamic knowledge. In: 2019 Fifteenth international conference on information processing (ICINPRO), pp 1–6. IEEE

  7. Dong L, Yang N, Wang W, Wei F, Liu X, Wang Y, Gao J, Zhou M, Hon H (2019) Unified language model pre-training for natural language understanding and generation. In: NeurIPS, pp 13042–13054

  8. Du X, Shao J, Cardie C (2017) Learning to ask: Neural question generation for reading comprehension. In: Association for computational linguistics. ACL (1)pp 1342–1352

  9. Fan A, Gardent C, Braud C, Bordes A (2019) Using local knowledge graph construction to scale seq2seq models to multi-document inputs. In: Association for computational linguistics. EMNLP/IJCNLP (1) pp 4184–4194

  10. Gao Y, Bing L, Chen W, Lyu MR, King I (2019) Difficulty controllable generation of reading comprehension questions. In: IJCAI, pp 4968–4974. http://www.ijcai.org

  11. Gao Y, Bing L, Li P, King I, Lyu MR (2019) Generating distractors for reading comprehension questions from real examinations. In: AAAI, pp 6423–6430. AAAI Press

  12. Guo Q, Kulkarni C, Kittur A, Bigham JP, Brunskill E (2016) Questimator: Generating knowledge assessments for arbitrary topics. In: IJCAI, pp 3726–3732. IJCAI/AAAI press

  13. Heilman M, Smith NA (2010) Good question! statistical ranking for question generation. In: The association for computational linguistics, HLT-NAACL. pp 609–617

  14. Jia X, Zhou W, Sun X, Wu Y (2021) EQG-RACE: Examination-type question generation. In: AAAI, pp 13143–13151. AAAI Press

  15. Kumar G, Banchs RE, D’Haro LF (2015) Revup: Automatic gap-fill question generation from educational texts. In: The association for computer linguistics, BEA@NAACL-HLT. pp 154–161

  16. Kumar V, Hua Y, Ramakrishnan G, Qi G, Gao L, Li Y (2019) Difficulty-controllable multi-hop question generation from knowledge graphs. In: ISWC (1). Lecture Notes in computer science, vol 11778. pp 82–398. Springer

  17. Lai G, Xie Q, Liu H, Yang Y, Hovy EH (2017) RACE: Large-scale reading comprehension dataset from examinations. In: Association for computational linguistics, EMNLP. pp 785–794

  18. Lelkes ÁD, Tran VQ, Yu C (2021) Quiz-style question generation for news stories. In: WWW. pp 2501–2511, ACM /IW3c2

  19. Li J, Luong M, Jurafsky D (2015) A hierarchical neural autoencoder for paragraphs and documents. In: ACL (1). pp 1106–1115, The association for computer linguistics

  20. Liang C, Yang X, Dave N, Wham D, Pursel B, Giles CL (2018) Distractor generation for multiple choice questions using learning to rank. In: BEA@NAACL-HLT. pp 284–290. Association for computational linguistics

  21. Liu B, Wei H, Niu D, Chen H, He Y (2020) Asking questions the human way: Scalable question-answer generation from text corpus. In: WWW, pp 2032–2043, ACM/IW3c2

  22. Liu B, Zhao M, Niu D, Lai K, He Y, Wei H, Xu Y (2019) Learning to generate questions by learningwhat not to generate. In: WWW, pp 1106–1118, ACM

  23. Luong T, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. In: EMNLP, pp 1412–1421, The association for computational linguistics

  24. Pennington J, Socher R, Manning CD (2014) Glove: Global vectors for word representation. In: EMNLP, pp 532–1543, ACL

  25. Peters ME, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. In: NAACL-HLT, pp 2227–2237, Association for computational linguistics

  26. Qi W, Yan Y, Gong Y, Liu D, Duan N, Chen J, Zhang R, Zhou M (2020) Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training. In: EMNLP (Findings). findings of ACL, vol. EMNLP 2020, pp 2401–2410, Association for Computational Linguistics

  27. Qiu Z, Wu X, Fan W (2020) Automatic distractor generation for multiple choice questions in standard tests. In: COLING, pp 2096–2106, International committee on computational linguistics

  28. Qu F, Jia X, Wu Y (2021) Asking questions like educational experts: Automatically generating question-answer pairs on real-world examination data. In: EMNLP (1). pp 2583–2593. Association for computational linguistics

  29. Ren S, Zhu KQ (2021) Knowledge-driven distractor generation for cloze-style multiple choice questions. In: AAAI pp 4339–4347, AAAI Press

  30. Sakaguchi K, Arase Y, Komachi M (2013) Discriminative approach to fill-in-the-blank quiz generation for language learners. In: ACL (2). pp 238–242, The association for computer linguistics

  31. Seo MJ, Kembhavi A, Farhadi A, Hajishirzi H (2017) Bidirectional attention flow for machine comprehension. In: ICLR, (Poster). Open Review Net

  32. Shah R, Shah D, Kurup L (2017) Automatic question generation for intelligent tutoring systems. In: 2017 2Nd international conference on communication systems, computing and IT applications (CSCITA). pp 127–132, IEEE

  33. Shuai P, Wei Z, Liu S, Xu X, Li L (2021) Topic enhanced multi-head co-attention: Generating distractors for reading comprehension. In: IJCNN, pp 1–8. IEEE

  34. Stasaski K, Hearst MA (2017) Multiple choice question generation utilizing an ontology. In: BEA@EMNLP pp 303–312, Association for computational linguistics

  35. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: NIPS. pp. 3104–3112

  36. Tan J, Wan X, Xiao J (2017) From neural sentence summarization to headline generation: a coarse-to-fine approach. In: IJCAI pp 4109–4115 https://www.ijcai.org

  37. Tang D, Duan N, Qin T, Zhou M (2017) Question answering and question generation as dual tasks. CoRR. arXiv:1706.02027

  38. Wagner C, Bolloju N (2005) Supporting knowledge management in organizations with conversational technologies: Discussion forums, weblogs, and wikis. J Database Manag 16(2):I

    Google Scholar 

  39. Wang W, Hao T, Liu W (2007) Automatic question generation for learning evaluation in medicine. In: ICWL, Lecture Notes in computer science, vol 4823. pp 242–251. Springer

  40. Xie J, Peng N, Cai Y, Wang T, Huang Q (2022) Diverse distractor generation for constructing high-quality multiple choice questions. IEEE ACM Trans Audio Speech Lang Process 30:280–291

    Article  Google Scholar 

  41. Yuan W, Yin H, He T, Chen T, Wang Q, Cui L (2022) Unified question generation with continual lifelong learning. In: WWW, pp 871–881, ACM

  42. Zhao Y, Ni X, Ding Y, Ke Q (2018) Paragraph-level neural question generation with maxout pointer and gated self-attention networks. In: EMNLP, pp 3901–3910, Association for computational linguistics

  43. Zhou Q, Yang N, Wei F, Tan C, Bao H, Zhou M (2017) Neural question generation from text: a preliminary study. In: NLPCC, Lecture Notes in computer science, vol 10619. pp 662–671. Springer

  44. Zhou X, Luo S, Wu Y (2020) Co-attention hierarchical network: Generating coherent long distractors for reading comprehension. In: AAAI, pp 9725–9732, AAAI Press

Download references

Acknowledgements

This work was supported by NSFC (grant No. 61877051). Li Li is the correspongding author for the paper.

Author information

Authors and Affiliations

  1. School of Computer & Information Science, Southwest University, Chongqing, 400700, Chongqing, China

    Pengju Shuai, Li Li & Sishun Liu

  2. School of Computer and Information Technology, University of Wollongong, Wollongong, 2522, New South Wales, Australia

    Jun Shen

Authors
  1. Pengju Shuai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Li Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sishun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jun Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Li Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shuai, P., Li, L., Liu, S. et al. QDG: A unified model for automatic question-distractor pairs generation. Appl Intell 53, 8275–8285 (2023). https://doi.org/10.1007/s10489-022-03894-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03894-6

Keywords

Search

Navigation