Skip to main content
SpringerLink
Log in

Impacts of ChatGPT-assisted writing for EFL English majors: Feasibility and challenges

  • Published:
Education and Information Technologies Aims and scope Submit manuscript

Abstract

Purpose

To determine the impacts of using ChatGPT to assist English as a foreign language (EFL) English college majors in revising essays and the possibility of leading to higher scores and potentially causing unfairness.

Design

A prospective, double-blinded, paired-comparison study was conducted in Feb. 2023. A total of 44 students provided 44 original essays and 44 ChatGPT-assisted revised essays, which were rated by two independent graders in a randomized and crossover fashion to minimize grading bias. The original and revision scores were paired for before-after comparison. Eight control essays were also rated by both graders to ensure inter-rater reliability.

Findings.

This study used a rigorous experimental design to confirm that ChatGPT-assisted revised essays led to significantly higher scores for EFL college English majors. Significant improvements were observed in all four dimensions of writing quality assessment, with the largest effects observed in vocabulary, followed by grammar, organization, and content. ChatGPT-assisted revised essays shifted the score curve from a normal distribution to a skewed distribution towards higher grades, with the greatest increase in revision scores seen among students who had lower original scores. This disproportionate improvement raises concerns about fairness in evaluation.

Value.

The findings suggest that ChatGPT is effective in providing timely feedback to EFL English majors in an affordable manner, but it also highlights the potential for unfairness in writing evaluation. We should note that ChatGPT-assisted revisions do not reveal learners’ writing competence. Therefore, new forms of writing performance assessment should be implemented in EFL composition classes in this AI era.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability

Data is available upon reasonable request to the corresponding author.

References

  • Carvalho, L., Martinez-Maldonado, R., Tsai, Y.-S., Markauskaite, L., & De Laat, M. (2022). How can we design for learning in an AI world? Computers and Education: Artificial Intelligence,3, 1–9. https://doi.org/10.1016/j.caeai.2022.100053

    Article  Google Scholar 

  • Cotton, D. R., Cotton, P. A., & Shipway, J. R. (2024). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 61(2), 228–239.

  • Fang, Y. (2010). Perceptions of the Computer-Assisted Writing Program among EFL College Learners. Journal of Educational Technology & Society,13(3), 246–256.

    Google Scholar 

  • Gao, C. A., Howard, F. M., Markov, N. S., Dyer, E. C., Ramesh, S., Luo, Y., & Pearson, A. T. (2023). Comparing scientific abstracts generated by ChatGPT to real abstracts with detectors and blinded human reviewers. NPJ Digital Medicine, 6(1), 75.

  • Godwin-Jones, R. (2022). Partnering with AI: Intelligent writing assistance and instructed language learning. Language Learning & Technology, 26(2), 5–24.

  • Henrickson, L. (2022). Cheating on your college essay with ChatGPT won’t get you good grades, say professors – but AI could make education fairer. Insider. https://www.businessinsider.com/professors-say-chatgpt-wont-kill-college-essays-make-education-fairer-2022-12. Accessed 17 Jan 2023.

  • Li, J. (2021). Research on AI-assisted hybrid teaching for English writing. In 2021 International Conference on Computers, Information Processing and Advanced Education (CIPAE), (pp. 309–312). IEEE.

  • Lim, W. M., Gunasekara, A., Pallant, J. L., Pallant, J. I., & Pechenkina, E. (2023). Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators. The International Journal of Management Education,21(2), 100790.

    Article  Google Scholar 

  • Qi, D. S., & Lapkin, S. (2001). Exploring the role of noticing in a three-stage second language writing task. Journal of Second Language Writing,10(4), 277–303.

    Article  Google Scholar 

  • Ranalli, J., & Yamashita, T. (2022). Automated written corrective feedback: Error-correction performance and timing of delivery. Language Learning & Technology,26(1), 1–25.

    Google Scholar 

  • Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6(1), 342–363.

  • Sachs, R., & Polio, C. (2007). Learners’ uses of two types of written feedback on a L2 writing revision task. Studies in Second Language Acquisition,29(1), 67–100.

    Article  Google Scholar 

  • Saricaglu, A. (2019). The impact of automated feedback on L2 learners’ written causal explanations. ReCALL,31(2), 189–203. https://doi.org/10.1017/S095834401800006X

    Article  Google Scholar 

  • Selber, S. (2022). Cheating on your college esssay with ChatGPT won’t get you good grades, say professors – but AI could make education fairer. Insider. https://www.businessinsider.com/professors-say-chatgpt-wont-kill-college-essays-make-education-fairer-2022-12. Accessed 16 Jan 2023.

  • Stacey, S. (2022). Cheating on your college essay with ChatGPT won’t get you good grades, say professors – but AI could make education fairer. Insider. https://www.businessinsider.com/professors-say-chatgpt-wont-kill-college-essays-make-education-fairer-2022-12. Accessed 16 Jan 2023.

  • Stokel-Walker, C. (2022). AI bot ChatGPT writes smart essays – should professors worry? Nature. https://doi.org/10.1038/d41586-022-04397-7

    Article  Google Scholar 

  • Straume, I., & Anson, C. (2022). Amazement and Trepidation: Implications of AI-Based Natural Language Production for the Teaching of Writing. Journal of Academic Writing,12(1), 1–9. https://doi.org/10.18552/joaw.v12i1.820

    Article  Google Scholar 

  • Susnjak, T. (2022). ChatGPT: The end of online exam integrity? arXiv. https://doi.org/10.48550/arXiv.2212.09292

  • Truscott, J., & Smith, M. S. (2011). Input, intake, and consciousness: The quest for a theoretical foundation. Studies in Second Language Acquisition,33(4), 497–528.

    Article  Google Scholar 

  • Zhai, X. (2022). ChatGPT user experience: Implications for education. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.4312418

  • Zomer, Z., & Frankenberg-Garcia, A. (2021). Beyong Grammar Error Correction: Improving L1-influenced research writing in English using pre-trained encoder-decoder models. In Findings of the Association for Computational Linguistics: EMNLP 2021 (pp. 2534–2540).

Download references

Author information

Authors and Affiliations

  1. Division of Urology, Department of Surgery, Far Eastern Memorial Hospital, New Taipei City, Taiwan

    Chung-You Tsai

  2. Department of Electrical Engineering, Yuan Ze University, Taoyuan City, Taiwan

    Chung-You Tsai

  3. Department of English, Tamkang University, New Taipei City, Taiwan

    Yi-Ti Lin & Iain Kelsall Brown

Authors
  1. Chung-You Tsai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yi-Ti Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Iain Kelsall Brown
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

C.Y. Tsai contributed to study design, data analysis, and the first draft of the manuscript, while Y.T. Lin contributed to data acquisition, discussion, and critical revision as a corresponding author. I. Brown contributed to data acquisition.

Corresponding author

Correspondence to Yi-Ti Lin.

Ethics declarations

The participants were protected in accordance to the ethics guidelines and the details were described in the section on participants.

Competing Interests

We, the authors, declare that we have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 377 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tsai, CY., Lin, YT. & Brown, I.K. Impacts of ChatGPT-assisted writing for EFL English majors: Feasibility and challenges. Educ Inf Technol (2024). https://doi.org/10.1007/s10639-024-12722-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10639-024-12722-y

Keywords

Search

Navigation