Skip to main content

Advertisement

SpringerLink
Log in

Question generation model based on key-phrase, context-free grammar, and Bloom’s taxonomy

  • Published:
Education and Information Technologies Aims and scope Submit manuscript

Abstract

Automated question generation is a task to generate questions from structured or unstructured data. The increasing popularity of online learning in recent years has given momentum to automated question generation in education field for facilitating learning process, learning material retrieval, and computer-based testing. This paper report on the development of question generation framework based on key-phrase method for online learning with a constraint that the generated questions should comply with the learning outcomes and skills from Bloom’s Taxonomy. The proposed method was tested using learning materials of Software Engineering course for undergraduate level written in Bahasa Indonesia obtained from Bina Nusantara’s (Binus’s) Online Learning repository. Using one-semester lecture material, this study generated 92,608 essay-type questions from 6-level Bloom’s Taxonomy which were further sampled randomly to obtain 120 question samples for method evaluation. Performance evaluation using average Bilingual Evaluation Understudy (BLEU) involving five independent reviewers toward samples of these questions achieved 0.921 and 0.6 Cohen’s Kappa. The relevance of Bloom’s Taxonomy level of the generated questions was evaluated by means of classification model with 0.99 accuracy. The results indicate that not only are the generated questions well understood and agreed by the reviewers, they are also relevant to the expected Bloom’s Taxonomy level there for the questions can be delivered to students in the respected course delivery and evaluation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Adamson, D., Bhartiya, D., Gujral, B., Kedia, R., Singh, A., & Rosé, C. P. (2013). Automatically generating discussion questions. In AIED (pp. 81–90). https://doi.org/10.1007/978-3-642-39112-5_9.

  • Al-Yahya, M. (2011). OntoQue: A question generation engine for educational assesment based on domain ontologies. In Proceedings of the 2011 11th IEEE International Conference on Advanced Learning Technologies, ICALT 2011 (pp. 393–395). IEEE. https://doi.org/10.1109/ICALT.2011.124.

  • Bloom, B.S. (1956). Taxonomy of educational objectives: The classification of educational goals.

  • Chao, Z., & Li, L. (2018). The combination of context information to enhance simple question answering. In Proceedings - 2018 5th international conference on behavioral, economic, and socio-cultural computing, BESC 2018 (pp. 109–114). https://doi.org/10.1109/BESC.2018.8697305.

    Chapter  Google Scholar 

  • Diab, S., & Sartawi, B. (2017). Classification of questions and learning outcome statements (LOS) into Bloom’s taxonomy (BT) by similarity measurements towards extracting of learning outcome from learning material. International Journal of Managing Information Technology, 9(2), 01–12. https://doi.org/10.5121/ijmit.2017.9201.

    Article  Google Scholar 

  • Divate, M., & Salgaonkar, A. (2017). Automatic question generation approaches and evaluation techniques. Current Science (00113891), 113(9).

  • Duan, N., Tang, D., Chen, P., & Zhou, M. (2017). Question generation for question answering. In EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings (pp. 877–885). https://doi.org/10.18653/v1/d17-1090.

  • Elsahar, H., Gravier, C., & Laforest, F. F. (2018). Zero-shot question generation from knowledge graphs for unseen predicates and entity types. NAACL-HLT, abs/1802.0, 218–228. https://doi.org/10.18653/v1/n18-1020.

  • Emu, I. H., Ahmed, A. U., Islam, M., Al Mamun, S., & Uddin, A. (2017). An efficient approach for Keyphrase extraction from English document. International Journal of Intelligent Systems and Applications, 9(12), 59–66. https://doi.org/10.5815/ijisa.2017.12.06.

    Article  Google Scholar 

  • Gan, W., & Yu, X. (2018). Automatic understanding and formalization of natural language geometry problems using syntax-semantics models. ICIC International, 14(1), 83–98.

    Google Scholar 

  • Gleason, N. W. (2018). Higher education in the era of the fourth industrial revolution. https://doi.org/10.1007/978-981-13-0194-0.

    Book  Google Scholar 

  • Graesser, A. C., Rus, V., Cai, Z., & Hu, X. (2009). Question answering and generation. Applied Natural Language Processing: Identification, Investigation and Resolution, 1, 1–16. https://doi.org/10.4018/978-1-60960-741-8.ch001.

    Article  Google Scholar 

  • Han, H., Yang, F., Huang, J., & Zhou, B. (2018). CFXGBoost: Topic phrase extraction based on context features and XGBoost for knowledge base question answering. In ICNC-FSKD 2017 - 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (pp. 2397–2402). IEEE. https://doi.org/10.1109/FSKD.2017.8393148.

  • Heilman, M., & Smith, N. A. (2010). Good question! Statistical ranking for question generation. NAACL HLT 2010 - Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Proceedings of the Main Conference, 609–617.

  • Heilman, M., Aleven, V., Cohen, W. W., Litman, D. J., & Smith, N. A. (2011). Automatic factual question generation from text. In Dissertation (p. 203). Retrieved from www.lti.cs.cmu.edu

  • Herranen, J., & Aksela, M. (2019). Student-question-based inquiry in science education. Studies in Science Education, 55(1), 1–36. https://doi.org/10.1080/03057267.2019.1658059.

    Article  Google Scholar 

  • Hopcroft, J. E., Motwani, R., & Ullman, J. D. (2001). Introduction to automata theory, languages, and computation, 2nd edition. ACM SIGACT News, 32, 60–65. https://doi.org/10.1145/568438.568455.

    Article  MATH  Google Scholar 

  • Huang, Y.-T., Chen, M. C., & Sun, Y. S. (2018). Bringing personalized learning into computer-aided question generation. CoRR, abs/1808.0. Retrieved from http://arxiv.org/abs/1808.09735

  • Jouault, C., Seta, K., & Hayashi, Y. (2016). Content-dependent question generation using LOD for history learning in open learning space. New Generation Computing, 34(4), 367–394. https://doi.org/10.1007/s00354-016-0404-x.

    Article  Google Scholar 

  • Kale, V. M., & Kiwelekar, A. W. (2013). An algorithm for question paper template generation in question paper generation system. In 2013 The International Conference on Technological Advances in Electrical, Electronics and Computer Engineering, TAEECE 2013 (pp. 256–261). https://doi.org/10.1109/TAEECE.2013.6557281.

  • Kumar, V., Ramakrishnan, G., & Li, Y.-F. (2018). A framework for automatic question generation from text using deep reinforcement learning. CoRR, abs/1808.0. Retrieved from http://arxiv.org/abs/1808.04961

  • Kurdi, G., Leo, J., Parsia, B., Sattler, U., & Al-Emari, S. (2019). A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education., 30, 121–204. https://doi.org/10.1007/s40593-019-00186-y.

    Article  Google Scholar 

  • Li, J., Gao, Y., Bing, L., King, I., & Lyu, M. R. (2019). Improving question generation with to the point context. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (pp. 3214–3224). Stroudsburg, PA, USA: Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1317.

  • Lindberg, D., & Sc, B. (2013). Automatic question generation from text for self-directed learning by.

  • Mandal, S., Mahata, S. K., & Das, D. (2018). Preparing Bengali-English Code-Mixed Corpus for Sentiment Analysis of Indian Languages. Retrieved from http://arxiv.org/abs/1803.04000

  • Meng, R., Zhao, S., Han, S., He, D., Brusilovsky, P., & Chi, Y. (2017). Deep Keyphrase generation. CoRR, abs/1704.0, 582–592. https://doi.org/10.18653/v1/P17-1054.

  • Mikolov, T., Chen, K., Corrado, G. S., Dean, J., Sutskever, I., Chen, K., … Dean, J. (2013). Distributed representations of words and phrases and their compositionality.

  • Moraes, S., Godbole, A., & Gharpure, P. (2018). Affinity analysis for context-free grammars. In IEEE International Conference on Power, Control, Signals and Instrumentation Engineering, ICPCSI 2017 (pp. 2471–2474). IEEE. https://doi.org/10.1109/ICPCSI.2017.8392161.

  • Nema, P., Mohankumar, A. K., Khapra, M. M., Srinivasan, B. V., & Ravindran, B. (2019). Let’s ask again: Refine network for automatic question generation. Retrieved from http://arxiv.org/abs/1909.05355

  • Quintana, R. M., & Tan, Y. (2019). Characterizing MOOC pedagogies: Exploring tools and methods for learning designers and researchers. Online Learning Journal, 23(4), 62–84. https://doi.org/10.24059/olj.v23i4.2084.

  • Rodrigues, H. P. (2017). Learning semantic patterns for question generation and question answering.

    Google Scholar 

  • Rus, V., Graesser, A., & Cai, Z. (2008). Question generation: Example of a multi-year evaluation campaign. Workshop on the Question Generation Shared Task and Evaluation Challenge, (January). Retrieved from http://www.cs.memphis.edu/~vrus/questiongeneration/5-RusEtAl-QG08.pdf

  • Singh, S., Tiwari, S., Varshney, A., & Sharma, A. (2017). Unsupervised key-phrase extraction using noun phrases. International Journal of Computer Applications, 162(1), 1–5. https://doi.org/10.5120/ijca2017913171.

    Article  Google Scholar 

  • Subramanian, S., Wang, T., Yuan, X., Zhang, S., Trischler, A., & Bengio, Y. (2018). Neural models for key phrase extraction and question generation. In QA@ACL (pp. 78–88). Association for Computational Linguistics. https://doi.org/10.18653/v1/w18-2609

  • Tong, P., Zhang, Q., & Yao, J. (2019). Leveraging domain context for question answering over knowledge graph. Data Science and Engineering, 4(4), 323–335. https://doi.org/10.1007/s41019-019-00109-w.

    Article  Google Scholar 

  • Turney, P. D. (2003). Coherent Keyphrase extraction via web mining. In IJCAI International Joint Conference on Artificial Intelligence.

  • Willis, A., Davis, G., Ruan, S., Manoharan, L., Landay, J., & Brunskill, E. (2019). Key phrase extraction for generating educational question-answer pairs. In Proceedings of the 6th 2019 ACM conference on learning at scale, L@S 2019. https://doi.org/10.1145/3330430.3333636.

  • Xie, Z., Zeng, Z., Zhou, G., & Wang, W. (2017). Topic enhanced deep structured semantic models for knowledge base question answering. SCIENCE CHINA Information Sciences, 60(11). https://doi.org/10.1007/s11432-017-9136-x.

  • Yang, Z., Hu, J., Salakhutdinov, R., & Cohen, W. W. (2017). Semi-supervised QA with generative domain-adaptive nets. In ACL 2017 - 55th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers). https://doi.org/10.18653/v1/P17-1096.

  • Yao, X., Bouma, G., Zhang, Y., Piwek, P., & Boyer, K. E. (2012). Semantic-based question generation and implementation. Dialogue & Discourse, 3(2), 11–42. https://doi.org/10.5087/dad.2012.202.

  • Ye, H., & Wang, L. (2018). Semi-supervised learning for Neural Keyphrase generation. Retrieved from http://arxiv.org/abs/1808.06773

Download references

Author information

Authors and Affiliations

  1. Computer Science Department, BINUS Graduate Program-Doctor of Computer Science, Bina Nusantara University, Jakarta, Indonesia

    Bambang Dwi Wijanarko, Yaya Heryadi & Widodo Budiharto

  2. Computer Science Department, BINUS Online Learning, Bina Nusantara University, Jakarta, Indonesia

    Bambang Dwi Wijanarko

  3. Faculty of Information Technology, Maranatha Christian University, Bandung, Indonesia

    Hapnes Toba

Authors
  1. Bambang Dwi Wijanarko
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yaya Heryadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hapnes Toba
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Widodo Budiharto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bambang Dwi Wijanarko.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wijanarko, B.D., Heryadi, Y., Toba, H. et al. Question generation model based on key-phrase, context-free grammar, and Bloom’s taxonomy. Educ Inf Technol 26, 2207–2223 (2021). https://doi.org/10.1007/s10639-020-10356-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10639-020-10356-4

Keywords

Search

Navigation