Abstract
This research investigates the impact of ChatGPT-generated feedback on the writing skills of first-year computing students at a Saudi University. Employing a qualitative research design, the study involved 111 male students, blinded to the switch from human to ChatGPT-generated feedback, ensuring unbiased reflections on their experiences. Over six weeks, students’ reactions to feedback were meticulously analyzed, revealing nuanced emotional, psychological, and educational impacts. The findings, organized into four distinct themes - Emotional and Psychological Responses, Perceived Quality and Usefulness, Progress and Development, and Feedback Content and Delivery - offer rich insights into the multifaceted experiences of students. While some students responded to the feedback provided during weeks 4 and 5 (ChatGPT-generated feedback), perceiving it as a catalyst for learning and self-improvement, others expressed concerns about its consistency and personalization. The study highlights the potential of ChatGPT in education, while also illuminating the need for a balanced, adaptive, and personalized approach to feedback that aligns with diverse learning styles, emotional responses, and educational needs.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The data supporting the findings of this research are not publicly available due to confidentiality agreements that protect the anonymity of the participants involved. To maintain this anonymity, the data set includes sensitive information that cannot be openly disclosed. However, upon a reasonable and justified request, the author may provide access to the data privately. Any such data sharing will be contingent upon a thorough review to ensure that all personally identifiable information, including students’ names or any other relevant details, has been appropriately redacted. This measure is taken to uphold the ethical standards of research and the privacy of individuals who participated in the study.
References
ABET. (2023). Criteria for Accrediting Computing Programs, 2023–2024. Accreditation board for engineering and technology. https://www.abet.org/accreditation/accreditation-criteria/criteria-for-accrediting-computing-programs-2023-2024/. Accessed 28 Sept 2023.
Abbas, N., Ali, I., Manzoor, R., Hussain, T., & HussainiM. H. A. (2023). Role of artificial intelligence tools in enhancing students’ educational performance at higher levels. Journal of Artificial Intelligence Machine Learning and Neural Network (JAIMLNN), 3(05). https://doi.org/10.55529/jaimlnn.35.36.49 (ISSN: 2799 − 1172).
ACM. (2020). Computing Curricula 2020: Paradigms for global computing education. Association for Computing Machinery.
Al Kandari, A. M., & Al Qattan, M. M. (2020). E-task-based learning approach to enhancing 21st-century learning outcomes. International Journal of Instruction, 13(1), 551–566.
Al-Bashir, M., Kabir, M., & Rahman, I. (2016). The value and effectiveness of feedback in improving students’ learning and professionalizing teaching in Higher Education. Journal of Education and Practice, 7(16), 38–41.
AlGhamdi, R. (2023). Development of soft skills among computing students in online task-based learning: Insights from technical communication course. International Journal of Technology in Education, 6(2). https://doi.org/10.46328/ijte.394
AlGhamdi, R., & Bahadad, A. (2018). Assessing the usages of LMS at KAU and Proposing FORCE Strategy for the Diffusion. Journal of King Abdulaziz University-Computing and Information Technology Sciences, 7(1). https://doi.org/10.4197/Comp.7-1.4
AlGhamdi, R. (2019). Fostering information technology students’ internship program | education and Information technologies. Education and Information Technologies, 24(5), 2727–2739. https://doi.org/10.1007/s10639-019-09891-6
Allen, M. (2016). Guide to e-Learning | Wiley Online Books (1st ed.). WILEY Online Library. Retrieved February 15, 2024, from https://onlinelibrary.wiley.com/doi/book/10.1002/9781119176268. Accessed 28 Sept 2023
Anthony, S., & Garner, B. (2016). Teaching soft skills to business students: An analysis of multiple pedagogical methods. Business and Professional Communication Quarterly, 79(3), 360–370. https://doi.org/10.1177/2329490616642247
Baidoo-Anu, D., & Owusu Ansah, L. (2023). Education in the era of Generative Artificial Intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning. Journal of AI, 7(1), 52–62. https://doi.org/10.2139/ssrn.4337484
Barrot, J. S. (2023). Using ChatGPT for second language writing: Pitfalls and potentials. Assessing Writing, 57, 100–145. https://doi.org/10.1016/j.asw.2023.100745
Baskara, F. R. (2023). Integrating ChatGPT into EFL writing instruction: Benefits and challenges. International Journal of Education and Learning, 5(1). https://doi.org/10.31763/ijele.v5i1.858
Belwal, R., Belwal, S., Sufian, A. B., & Al Badi, A. (2020). Project-based learning (PBL): Outcomes of students’ engagement in an external consultancy project in Oman. Education + Training, 63(3), 336–359. https://doi.org/10.1108/ET-01-2020-0006
Berkhof, M., van Rijssen, H. J., Schellart, A. J. M., Anema, J. R., & van der Beek, A. J. (2011). Effective training strategies for teaching communication skills to physicians: An overview of systematic reviews. Patient Education and Counseling, 84(2), 152–162. https://doi.org/10.1016/j.pec.2010.06.010
Cao, S., & Zhong, L. (2023). Exploring the effectiveness of ChatGPT-based feedback compared with teacher feedback and self-feedback: Evidence from Chinese to English translation (arXiv:2309.01645). arXiv. https://doi.org/10.48550/arXiv.2309.01645
Chan, C. K. Y., & Hu, W. (2023). Students’ voices on generative AI: Perceptions, benefits, and challenges in higher education. International Journal of Educational Technology in Higher Education, 20(1), 20–43. https://doi.org/10.1186/s41239-023-00411-8
Dai, W., Lin, J., Jin, H., Li, T., Tsai, Y. S., Gašević, D., & Chen, G. (2023). Can large language models provide feedback to students? A case study on ChatGPT. IEEE International Conference on Advanced Learning Technologies (ICALT), 2023, 323–325. https://doi.org/10.1109/ICALT58122.2023.00100
Dirksen, J. (2015). Design for how people learn (2nd ed.). New Riders.
Firat, M. (2023). What ChatGPT means for universities: Perceptions of scholars and students. Journal of Applied Learning and Teaching, 6(1). https://doi.org/10.37074/jalt.2023.6.1.22
Ion, G., Cano-García, E., & Fernández-Ferrer, M. (2017). Enhancing self-regulated learning through using written feedback in higher education. International Journal of Educational Research, 85, 1–10. https://doi.org/10.1016/j.ijer.2017.06.002
Jackson, K., & Bazeley, P. (2019). Qualitative data analysis with NVivo (3rd ed.). Sage Publication Ltd. https://uk.sagepub.com/en-gb/eur/qualitative-data-analysis-with-nvivo/book261349. Accessed 28 Sept 2023.
Jonsson, A. (2013). Facilitating productive use of feedback in higher education. Active Learning in Higher Education, 14(1), 63–76. https://doi.org/10.1177/1469787412467125
Katz, A., Wei, S., Nanda, G., Brinton, C., & Ohland, M. (2023). Exploring the efficacy of ChatGPT in analyzing student teamwork feedback with an existing taxonomy (Version 1). arXiv. https://doi.org/10.48550/ARXIV.2305.11882
KAU E-Learning Deanship. (2014). Adopting the Blackboard system in the management of the educational process at the university. https://www.kau.edu.sa/Pages-Blackboard.aspx. Accessed 28 Sept 2023.
Kim, A. Y. (Alicia), & Kim, H. J. (Eds.). (2017). The effectiveness of instructor feedback for learning-oriented language assessment: Using an integrated reading-to-write task for English for academic purposes. Assessing Writing, 32, 57–71. https://doi.org/10.1016/j.asw.2016.12.001
Kuleto, V., Ilić, M., Dumangiu, M., Ranković, M., Martins, O. M. D., Păun, D., & Mihoreanu, L. (2021). Exploring opportunities and challenges of artificial intelligence and machine learning in higher education institutions. Sustainability, 13(18), 10424. https://doi.org/10.3390/su131810424. Article 18.
Lee, A. V. Y. (2023). Supporting students’ generation of feedback in large-scale online course with artificial intelligence-enabled evaluation. Studies in Educational Evaluation, 77, 101250. https://doi.org/10.1016/j.stueduc.2023.101250
Miao, F., & Holmes, W. (2023). Guidance for generative AI in education and research. UNESCO Digital Library. https://unesdoc.unesco.org/ark:/48223/pf0000386693. Accessed 28 Sept 2023.
Nazari, N., Shabbir, M. S., & Setiawan, R. (2021). Application of artificial intelligence powered digital writing assistant in higher education: Randomized controlled trial. Heliyon, 7(5), e07014. https://doi.org/10.1016/j.heliyon.2021.e07014
Olsen, T., & Hunnes, J. (2023). Improving students’ learning—the role of formative feedback: Experiences from a crash course for business students in academic writing. Assessment and Evaluation in Higher Education, 1–13. https://doi.org/10.1080/02602938.2023.2187744
Rahman, M. M. (2010). Teaching oral communication skills: A task-based approach. ESP world, 9(1), 1–11.
Rahman, M. M., & Watanobe, Y. (2023). ChatGPT for education and research: Opportunities, threats, and strategies. Applied Sciences, 13(9), 5783. https://doi.org/10.3390/app13095783
Rezk, Y. (2021). Promoting students’ learning through constructive feedback strategies [PhD, Northeastern University]. https://repository.library.northeastern.edu/files/neu:bz60w834v. Accessed 28 Sept 2023.
Ritter, B. A., Small, E. E., Mortimer, J. W., & Doll, J. L. (2018). Designing management curriculum for workplace readiness: Developing students’ soft skills. Journal of Management Education, 42(1), 80–103. https://doi.org/10.1177/1052562917703679
Rodríguez-Bonces, M., & Rodríguez-Bonces, J. (2010). Task-based language learning: Old approach, new style. A new lesson to learn. Profile Issues in Teachers` Professional Development, 12(2), 165–178.
Roessingh, H. (2014). Teachers’ roles in designing meaningful tasks for mediating language learning through the use of ICT: A reflection on authentic learning for young ELLs. Canadian Journal of Learning and Technology, 40(1), 1–24. https://doi.org/10.21432/T2PP4M
Seo, K., Tang, J., Roll, I., Fels, S., & Yoon, D. (2021). The impact of artificial intelligence on learner–instructor interaction in online learning. International Journal of Educational Technology in Higher Education, 18(1), 1–23. https://doi.org/10.1186/s41239-021-00292-9
Seufert, S., Burkhard, M., & Handschuh, S. (2021). Fostering students’ academic writing skills: Feedback model for an AI-enabled support environment. Innovate Learning Summit. https://www.alexandria.unisg.ch/handle/20.500.14171/109740. Accessed 28 Sept 2023.
Shaik, T., Tao, X., Li, Y., Dann, C., McDonald, J., Redmond, P., & Galligan, L. (2022). A review of the trends and challenges in adopting natural language processing methods for education feedback analysis. IEEE Access : Practical Innovations, Open Solutions, 10, 56720–56739. https://doi.org/10.1109/ACCESS.2022.3177752
Shaikh, S., Yayilgan, S. Y., Klimova, B., & Pikhart, M. (2023). Assessing the usability of ChatGPT for formal English Language Learning. European Journal of Investigation in Health Psychology and Education, 13(9), 1937–1960. https://doi.org/10.3390/ejihpe13090140
Su, Y., Lin, Y., & Lai, C. (2023). Collaborating with ChatGPT in argumentative writing classrooms. Assessing Writing, 57, 100752. https://doi.org/10.1016/j.asw.2023.100752
Tong, S., Jia, N., Luo, X., & Fang, Z. (2021). The Janus face of artificial intelligence feedback: Deployment versus disclosure effects on employee performance. Strategic Management Journal, 42(9), 162–171. https://doi.org/10.1002/smj.3322
van den Branden, K. (Ed.). (2006). Task-based language education: From theory to practice. Cambridge University Press. https://doi.org/10.1017/CBO9780511667282
Walker, H. M. (2018). Teaching Computing: A practitioner’s perspective. CRC Press. https://doi.org/10.1201/9781315269108
Willis, J., & Willis, D. (2007). Doing task-based teaching—Oxford handbooks for Language teachers (1st ed.). Oxford University Press.
Xiao, Y., & Zhi, Y. (2023). An exploratory study of EFL Learners’ Use of ChatGPT for Language Learning tasks: Experience and perceptions. Languages, 8(3), 212. https://doi.org/10.3390/languages8030212
Yu, S. (2021). Feedback-giving practice for L2 writing teachers: Friend or foe? Journal of Second Language Writing, 52, 100798. https://doi.org/10.1016/j.jslw.2021.100798
Acknowledgements
The author acknowledges the assistance of OpenAI’s ChatGPT-4 in refning the language, proofreading, and enhancing the structural coherence of the manuscript.
Funding
The Deanship of Scientific Research (DSR) at King Abdulaziz University (KAU), Jeddah, Saudi Arabia has funded this project under grant no (G: 327-611-1443).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The author declares that there is no conflict of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
AlGhamdi, R. Exploring the impact of ChatGPT-generated feedback on technical writing skills of computing students: A blinded study. Educ Inf Technol (2024). https://doi.org/10.1007/s10639-024-12594-2
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10639-024-12594-2