Abstract
While open-ended self-explanations have been shown to promote robust learning in multiple studies, they pose significant challenges to automated grading and feedback in technology-enhanced learning, due to the unconstrained nature of the students’ input. Our work investigates whether recent advances in Large Language Models, and in particular ChatGPT, can address this issue. Using decimal exercises and student data from a prior study of the learning game Decimal Point, with more than 5,000 open-ended self-explanation responses, we investigate ChatGPT's capability in (1) solving the in-game exercises, (2) determining the correctness of students’ answers, and (3) providing meaningful feedback to incorrect answers. Our results showed that ChatGPT can respond well to conceptual questions, but struggled with decimal place values and number line problems. In addition, it was able to accurately assess the correctness of 75% of the students’ answers and generated generally high-quality feedback, similar to human instructors. We conclude with a discussion of ChatGPT's strengths and weaknesses and suggest several venues for extending its use cases in digital teaching and learning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adams, D.M., Clark, D.B.: Integrating self-explanation functionality into a complex game environment: keeping gaming in motion. Comput. Educ. 73, 149–159 (2014)
Bengio, Y., Ducharme, R., Vincent, P.: A neural probabilistic language model. Adv. Neural Inform. Process. Syst. 13 (2000)
Bubeck, S., et al.: Sparks of artificial general intelligence: early experiments with gpt-4. arXiv preprint arXiv:2303.12712 (2023)
Corral, D., Carpenter, S.K., Clingan-Siverly, S.: The effects of immediate versus delayed feedback on complex concept learning. Quar. J. Exper. Psychol. 74(4), 786–799 (2021)
Cotton, D.R.E., Cotton, P.A., Shipway, J.R.: Chatting and Cheating: Ensuring academic integrity in the era of ChatGPT. EdArXiv, 1–11 (2023)
DeCuir-Gunby, J.T., Marshall, P.L., McCulloch, A.W.: Developing and using a codebook for the analysis of interview data: an example from a professional development research project. Field Meth. 23(2), 136–155 (2011)
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Frieder, S., et al.: Mathematical capabilities of chatgpt. arXiv preprint arXiv:2301.13867 (2023)
Hou, X., Nguyen, H.A., Richey, J.E., Harpstead, E., Hammer, J., McLaren, B.M.: Assessing the effects of open models of learning and enjoyment in a digital learning game. Int. J. Artif. Intell. Educ. 32(1), 120–150 (2022)
Hou, X., Nguyen, H.A., Elizabeth Richey, J., McLaren, B. M.: Exploring how gender and enjoyment impact learning in a digital learning game. In: Bittencourt, I.I., Cukurova, M., Muldner, K., Luckin, R., Millán, E. (eds.) Artificial Intelligence in Education: 21st International Conference, AIED 2020, Ifrane, Morocco, July 6–10, 2020, Proceedings, Part I, pp. 255–268. Springer International Publishing, Cham (2020). https://doi.org/10.1007/978-3-030-52237-7_21
Hsu, Chung-Yuan., Tsai, Chin-Chung.: Investigating the impact of integrating self-explanation into an educational game: A pilot study. In: Chang, Maiga, Hwang, Wu-Yuin., Chen, Ming-Puu., Müller, Wolfgang (eds.) Edutainment 2011. LNCS, vol. 6872, pp. 250–254. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23456-9_49
Johnson, C.I., Mayer, R.E.: Applying the self-explanation principle to multimedia learning in a computer-based game-like environment. Comput. Hum. Behav. 26(6), 1246–1252 (2010)
Kulhavy, R.W., Stock, W.A.: Feedback in written instruction: the place of response certitude. Educ. Psychol. Rev. 1, 279–308 (1989)
Kung, T.H., et al.: Performance of ChatGPT on USMLE: potential for AI-assisted medical education using large language models. PLoS Digital Health 2, 2, e0000198 (2023)
McLaren, B.M., Adams, D.M., Mayer, R.E., Forlizzi, J.: A computer-based game that promotes mathematics learning more than a conventional approach. Int. J. Game-Bas. Learn. 7(1), 36–56 (2017)
McLaren, B.M., DeLeeuw, K.E., Mayer, R.E.: Polite web-based intelligent tutors: can they improve learning in classrooms? Comput. Educ. 56(3), 574–584 (2011)
McLaren, B.M., Richey, J.E., Nguyen, H.A., Mogessie, M.: Focused self-explanations lead to the best learning outcomes in a digital learning game. In: Proceedings of the 17th International Conference of the Learning Sciences, pp. 1229–1232 ISLS (2022)
Moore, S., Nguyen, H.A., Bier, N., Domadia, T., Stamper, J.: Assessing the quality of student-generated short answer questions using GPT-3. In: Educating for a New Future: Making Sense of Technology-Enhanced Learning Adoption: 17th European Conference on Technology Enhanced Learning, EC-℡ 2022, Toulouse, France, September 12–16, 2022, Proceedings, pp. 243–257. Springer (2022)
Nguyen, H., Hou, X., Stamper, J., McLaren, B.M.: Moving beyond test scores: analyzing the effectiveness of a digital learning game through learning analytics. In: Proceedings of the 13th International Conference on Educational Data Mining (2020)
Nguyen, H.A., Bhat, S., Moore, S., Bier, N., Stamper, J.: Towards generalized methods for automatic question generation in educational domains. In: Educating for a New Future: Making Sense of Technology-Enhanced Learning Adoption: 17th European Conference on Technology Enhanced Learning, EC-℡ 2022, Toulouse, France, September 12–16, 2022, Proceedings. pp. 272–284. Springer (2022)
Nguyen, H.A., Hou, X., Stec, H., Di, S., Stamper, J., McLaren, B.: Examining the benefits of prompted self-explanation for problem-solving in a decimal learning game. In: Proceedings of the International Conference on Artificial Intelligence in Education. Springer
O’Neil, H.F., Chung, G.K., Kerr, D., Vendlinski, T.P., Buschang, R.E., Mayer, R.E.: Adding self-explanation prompts to an educational computer game. Comput. Hum. Behav. 30, 23–28 (2014)
Pardos, Z.A., Bhandari, S.: Learning gain differences between ChatGPT and human tutor generated algebra hints. arXiv preprint arXiv:2302.06871 (2023)
Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., Yang, D.: Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476 (2023)
Ramesh, D., Sanampudi, S.K.: An automated essay scoring systems: a systematic literature review. Artif. Intell. Rev. 55(3), 2495–2527 (2022)
Razzaq, R., Ostrow, K.S., Heffernan, N.T.: Effect of immediate feedback on math achievement at the high school level. In: Bittencourt, I.I., Cukurova, M., Muldner, K., Luckin, R., Millán, E. (eds.) AIED 2020. LNCS (LNAI), vol. 12164, pp. 263–267. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-52240-7_48
Richey, J.E., et al.: More confusion and frustration, better learning: The impact of erroneous examples. Comput. Educ. 139, 173–190 (2019)
Rudolph, J., Tan, S., Tan, S.: ChatGPT: bullshit spewer or the end of traditional assessments in higher education? J. Appl. Learn. Teach. 6, 1 (2023)
Swart, E.K., Nielen, T.M., Sikkema-de Jong, M.T.: Supporting learning from text: a meta-analysis on the timing and content of effective feedback. Educ. Res. Rev. 28, 100296 (2019)
Van der Kleij, F.M., Feskens, R.C., Eggen, T.J.: Effects of feedback in a computer-based learning environment on students’ learning outcomes: a meta-analysis. Rev. Educ. Res. 85(4), 475–511 (2015)
Wylie, R., Chi, M.T.: The self-explanation principle in multimedia learning. The Cambridge Handbook of Multimedia Learning, vol. 413 (2014)
Ye, J., et al.: A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Models. arXiv preprint arXiv:2303.10420 (2023)
Zhu, M., Lee, H.-S., Wang, T., Liu, O.L., Belur, V., Pallant, A.: Investigating the impact of automated feedback on students’ scientific argumentation. Int. J. Sci. Educ. 39(12), 1648–1668 (2017)
Acknowledgements
This work was supported by NSF Award #DRL-2201796. Thanks to Injila Adil for assisting with the coding of the self-explanation feedback.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Nguyen, H.A., Stec, H., Hou, X., Di, S., McLaren, B.M. (2023). Evaluating ChatGPT’s Decimal Skills and Feedback Generation in a Digital Learning Game. In: Viberg, O., Jivet, I., Muñoz-Merino, P., Perifanou, M., Papathoma, T. (eds) Responsive and Sustainable Educational Futures. EC-TEL 2023. Lecture Notes in Computer Science, vol 14200. Springer, Cham. https://doi.org/10.1007/978-3-031-42682-7_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-42682-7_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-42681-0
Online ISBN: 978-3-031-42682-7
eBook Packages: Computer ScienceComputer Science (R0)