Skip to main content
SpringerLink
Log in

The unified difficulty ranking mechanism for automatic multiple choice question generation in digital storytelling domain

  • Published:
Education and Information Technologies Aims and scope Submit manuscript

Abstract

Multiple Choice Questions (MCQs) are an important evaluation technique for both examinations and learning activities. However, the manual creation of questions is time-consuming and challenging for teachers. Hence, there is a notable demand for an Automatic Question Generation (AQG) system. Several systems have been created for this aim, but the generated questions failed to meet the requirements for student assessment effectively. Consequently, research in education technology, natural language processing, and KG development technology to support the AQG systems is still in its infancy. In this paper, the innovative integrated AQG framework for creating MCQs with difficulty levels is presented. The improved KG is first built as a source for question generation through corresponding queries. The distractors or wrong choices are generated by proposing a unified difficulty ranking mechanism, which includes the hybrid technique of WordNet-based and Linked Data (LD)-based semantic similarity together with property filtering score. Furthermore, the syntactic feature i.e. part-of-speech is utilized for the best distractors generation. The experimental results of the proposed unified difficulty ranking mechanism demonstrate an accuracy of 94% on the KG test dataset and 75% on public datasets. The accuracy of distractors’ correctness is 60% on the test dataset while it is 72% on the public dataset. The results highlight the efficiency of the proposed unified difficulty ranking mechanism for automatic MCQs generation in the digital storytelling domain.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data availability

The datasets generated during and analyzed during the current study are available from the corresponding author upon reasonable request.

Notes

  1. https://wordnet.princeton.edu/

  2. https://www.dbpedia.org/

  3. https://www.w3.org/TR/rdf-sparql-query/

  4. https://neo4j.com/

  5. https://neo4j.com/developer/cypher/

  6. https://neo4j.com/

  7. https://wordnet.princeton.edu/

  8. https://www.nltk.org/

  9. https://neo4j.com/labs/neosemantics/4.0/

References

  • Akben, N. (2020). Effects of the problem-posing approach on students’ problem solving skills and metacognitive awareness in science education. Research in Science Education, 50(3), 1143–1165. https://doi.org/10.1007/s11165-018-9726-7

    Article  Google Scholar 

  • Alfarhood, S., Gauch, S., & Labille, K. (2019). Semantic distance spreading across entities in linked open data. Information, 10(1), 15. https://doi.org/10.3390/info10010015

    Article  Google Scholar 

  • Almuayqil, S., Abd El-Ghany, S., & Shehab, A. (2020). Towards an ontology-based fully integrated system for student e-assessment. Journal of Theoretical and Applied Information Technology, 98(21), 3514–3527.

    Google Scholar 

  • Alsubait, T., Parsia, B., & Sattler, U. (2013). A similarity-based theory of controlling mcq difficulty. 2013 second international conference on e-learning and e-technologies in education (iceee) (pp. 283–288).

  • BS, B. (1956). Taxonomy of educational objectives, handbook i: the cognitive domain. New York: David McKay Co Inc, New York.

  • Cubric, M., & Tosic, M. (2020). Design and evaluation of an ontology-based tool for generating multiple-choice questions. Interactive Technology and Smart Education, 17(2), 109–131. https://doi.org/10.1108/ITSE-05-2019-0023

    Article  Google Scholar 

  • Das, B., Majumder, M., Phadikar, S., & Sekh, A. A. (2021). Multiple-choice question generation with auto-generated distractors for computer-assisted educational assessment. Multimedia Tools and Applications, 80(21–23), 31907–31925. https://doi.org/10.1007/s11042-021-11222-2

    Article  Google Scholar 

  • Demaidi, M. N., Gaber, M. M., & Filer, N. (2017). Evaluating the quality of the ontology-based auto-generated questions. Smart Learning Environments,4, 1–24. https://doi.org/0.1186/s40561-017-0046-6

  • Faizan, A., & Lohmann, S. (2018). Automatic generation of multiple choice questions from slide content using linked data. Proceedings of the 8th international conference on web intelligence, mining and semantics (pp. 1–8).

  • Fattoh, I., Aboutable, A., & Haggag, M. (2014). Sematic attributes model for automatic generation of multiple choice questions. International Journal of Compute Applications, 103(1), 18–24. https://doi.org/10.5120/18038-8544

    Article  Google Scholar 

  • Franzoni, V., Lepri, M., & Milani, A. (2019). Topological and semantic graph-based author disambiguation on dblp data in neo4j. arXiv:1901.08977, https://doi.org/10.48550/arXiv.1901.08977

  • Fu, C., Liu, W., Xu, J., Xu, J., & Cheng, W. (2021). Enhanced knowledge graph for true-false question generation: A case study in computer networks course. 2021 3rd international workshop on artificial intelligence and education (waie) (pp. 17–21).

  • Han, M., Zhang, X., Yuan, X., Jiang, J., Yun, W., & Gao, C. (2021). A survey on the techniques, applications, and performance of short text semantic similarity. Concurrency and Computation: Practice and Experience, 33(5), e5971. https://doi.org/10.1002/cpe.5971

    Article  Google Scholar 

  • Ibrahim Teo, N. H., & Joy, M. (2017). Evaluation of an automatic question generation approach using ontologies. Proceedings of the 15th european conference on e-learning (ecel 2016) (pp. 735–743).

  • Jatnika, D., Bijaksana, M. A., & Suryani, A. A. (2019). Word2vec model analysis for semantic similarities in english words. Procedia Computer Science, 157, 160–167. https://doi.org/10.1016/j.procs.2019.08.153

    Article  Google Scholar 

  • Joshi, S., Shah, P., & Shah, S. (2021). Automatic question paper generation, according to bloom’s taxonomy, by generating questions from text using natural language processing. International Research Journal of Engineering and Technology, 6, 495–501.

    Google Scholar 

  • Kurdi, G., Leo, J., Parsia, B., Sattler, U., & Al-Emari, S. (2020). A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education, 30, 121–204. https://doi.org/10.1007/s40593-019-00186-y

    Article  Google Scholar 

  • Kusuma, S. F., Siahaan, D. O., & Fatichah, C. (2022). Automatic question generation with various difficulty levels based on knowledge ontology using a query template. Knowledge-Based Systems, 249, 108906. https://doi.org/10.1016/j.knosys.2022.108906

    Article  Google Scholar 

  • Le, N. T. & Pinkwart, N. (2014). Question generation using wordnet. Proceedings of the 22nd international conference on computers in education. Asia-Pacific Society for Computers in Education.

  • Lei, Q., & Liu, Y. (2019). Constructing movie domain knowledge graph based on lod. 2019 twelfth international conference on ubi-media computing (ubi-media) (pp. 54–57).

  • Leo, J., Kurdi, G., Matentzoglu, N., Parsia, B., Sattler, U., Forge, S., ... Dowling, W. (2019). Ontology-based generation of medical, multi-term mcqs. International Journal of Artificial Intelligence in Education,29, 145–188. https://doi.org/10.1007/s40593-018-00172-w

  • Lin, C., Liu, D., Pang, W., & Wang, Z. (2015). Sherlock: A semi-automatic framework for quiz generation using a hybrid semantic similarity measure. Cognitive computation, 7, 667–679. https://doi.org/10.1007/s12559-015-9347-7

    Article  Google Scholar 

  • Ma, L., & Ma, Y. (2019). Automatic question generation based on mooc video subtitles and knowledge graph. Proceedings of the 2019 7th international conference on information and education technology (pp. 49–53).

  • Maheen, F., Asif, M., Ahmad, H., Ahmad, S., Alturise, F., Asiry, O., & Ghadi, Y. Y. (2022). Automatic computer science domain multiple-choice questions generation based on informative sentences. PeerJ Computer Science, 8, e1010. https://doi.org/10.7717/peerj-cs.1010

    Article  Google Scholar 

  • Meng, L., Huang, R., & Gu, J. (2013). A review of semantic similarity measures in wordnet. International Journal of Hybrid Information Technology, 6(1), 1–12.

    Google Scholar 

  • Meymandpour, R., & Davis, J. G. (2016). A semantic similarity measure for linked data: An information content-based approach. Knowledge-Based Systems, 109, 276–293. https://doi.org/10.1016/j.knosys.2016.07.012

    Article  Google Scholar 

  • Nimkanjana, K., & Witosurapot, S. (2018a). A simple approach for enabling sparql-based temporal queries for media fragments. Proceedings of the 2018 7th international conference on software and computer applications (pp. 212–216).

  • Nimkanjana, K., & Witosurapot, S. (2018b). Video-based question generation for mobile learning. Proceedings of the 2nd international conference on education and multimedia technology (pp. 5–8).

  • Nuthong, S., & Witosurapot, S. (2017). Enabling fine granularity of difficulty ranking measure for automatic quiz generation. 2017 9th international conference on information technology and electrical engineering (icitee) (pp. 1–6).

  • Passant, A. (2010). Measuring semantic distance on linking data and using it for resources recommendations. 2010 aaai spring symposium series.

  • Patra, R., & Saha, S. K. (2019). A hybrid approach for automatic generation of named entity distractors for multiple choice questions. Education and Information Technologies, 24, 973–993. https://doi.org/10.1007/s10639-018-9814-3

    Article  Google Scholar 

  • Peng, C., Xia, F., Naseriparsa, M., & Osborne, F. (2023). Knowledge graphs: Opportunities and challenges. Artificial Intelligence Review, 1–32. https://doi.org/10.1007/s10462-023-10465-9

  • Rocha, O. R., & Zucker, C. F. (2017). Automatic generation of educational quizzes from domain ontologies. Edulearn 2017-9th international conference on education and new learning technologies (pp. 4024–4030).

  • Scharpf, P., Schubotz, M., Spitz, A., Greiner-Petter, A., & Gipp, B. (2022). Collaborative and ai-aided exam question generation using wikidata in education. arXiv:2211.08361. https://doi.org/10.13140/RG.2.2.30988.18568

  • Slimani, T. (2013). Description and evaluation of semantic similarity measures approaches. arXiv:1310.8059

  • Sokhanvar, Z., Salehi, K., & Sokhanvar, F. (2021). Advantages of authentic assessment for improving the learning experience and employability skills of higher education students: A systematic literature review. Studies in Educational Evaluation, 70, 101030. https://doi.org/10.1016/j.stueduc.2021.101030

    Article  Google Scholar 

  • Sunilkumar, P., & Shaji, A. P. (2019). A survey on semantic similarity. 2019 international conference on advances in computing, communication and control (icac3) (pp. 1–8).

  • Tan, J., Qiu, Q., Guo, W., & Li, T. (2021). Research on the construction of a knowledge graph and knowledge reasoning model in the field of urban traffic. Sustainability, 13(6), 3191. https://doi.org/10.3390/su13063191

    Article  Google Scholar 

  • Vega-Gorgojo, G. (2019). Clover quiz: A trivia game powered by dbpedia. Semantic Web, 10(4), 779–793. https://doi.org/10.3233/SW-180326

    Article  Google Scholar 

  • Wang, H.-C., Chiang, Y.-H., & Chen, I.-F. (2023). A method for generating course test questions based on natural language processing and deep learning. Education and Information Technologies, 1–23. https://doi.org/10.1007/s10639-023-12159-9

  • Wang, H.-C., Maslim, M., & Kan, C.-H. (2023). A question-answer generation system for an asynchronous distance learning platform. Education and Information Technologies, 1–30. https://doi.org/10.1007/s10639-023-11675-y

  • Wijanarko, B. D., Heryadi, Y., Toba, H., & Budiharto, W. (2021). Question generation model based on key-phrase, context-free grammar, and bloom’s taxonomy. Education and Information Technologies, 26, 2207–2223. https://doi.org/10.1007/s10639-020-10356-4

    Article  Google Scholar 

  • Zhu, L., Xu, Z., Yang, Y., & Hauptmann, A. G. (2017). Uncovering the temporal context for video question answering. International Journal of Computer Vision, 124, 409–421. https://doi.org/10.1007/s11263-017-1033-7

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgements

This work was supported by the Higher Education Research Promotion and the Tailand’s Education Hub for Southern Region of ASEAN Countries Project Office of the Higher Education Commission.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Faculty of Engineering, Prince of Songkla University, Kanjanavanich, Hatyai, Songkhla, 90110, Thailand

    Lae Lae Shwe & Suntorn Witosurapot

  2. Innovation and Digital Transformation, Department of Interdisciplinary Engineering, Faculty of Engineering, Prince of Songkla University, Kanjanavanich, Hatyai, Songkhla, 90110, Thailand

    Sureena Matayong

Authors
  1. Lae Lae Shwe
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sureena Matayong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Suntorn Witosurapot
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sureena Matayong.

Ethics declarations

Conflicts of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shwe, L.L., Matayong, S. & Witosurapot, S. The unified difficulty ranking mechanism for automatic multiple choice question generation in digital storytelling domain. Educ Inf Technol (2024). https://doi.org/10.1007/s10639-024-12666-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10639-024-12666-3

Keywords

Search

Navigation