The Crutch or the Ceiling? How Different Generations of LLMs Shape EFL Student Writings
Hengky Susanto, David James Woo, Chingyi Yeung, Stephanie Wing Yan Lo-Philip, Chi Ho Yeung
TLDR
This study investigates how different generations of LLMs impact EFL student writing, finding advanced models boost scores but may mask true ability.
Key contributions
- Analyzes how LLMs, pre- and post-ChatGPT, affect secondary EFL student writing development.
- Finds advanced LLMs improve scores and lexical diversity for lower-proficiency EFL learners.
- Reveals increased LLM assistance negatively correlates with expert ratings, indicating surface fluency.
- Proposes pedagogical shifts to verify learning processes and differentiate AI functions for genuine learning.
Why it matters
This paper is crucial for educators using LLMs, highlighting that advanced models can mask true student ability. It urges a pedagogical shift to focus on the learning process, ensuring AI fosters genuine skill development rather than just surface-level fluency.
Original Abstract
The rapid evolution of Large Language Models (LLMs) has made them powerful tools for enhancing student writing. This study explores the extent and limitations of LLMs in assisting secondary-level English as a Foreign Language (EFL) students with their writing tasks. While existing studies focus on output quality, our research examines the developmental shift in LLMs and their impact on EFL students, assessing whether smarter models act as true scaffolds or mere compensatory crutches. To achieve this, we analyse student compositions assisted by LLMs before and after ChatGPT's release, using both expert qualitative scoring and quantitative metrics (readability tests, Pearson's correlation coefficient, MTLD, and others). Our results indicate that advanced LLMs boost assessment scores and lexical diversity for lower-proficiency learners, potentially masking their true ability. Crucially, increased LLM assistance correlated negatively with human expert ratings, suggesting surface fluency without deep coherence. To transform AI-assisted practice into genuine learning, pedagogy must shift from focusing on output quality to verifying the learning process. Educators should align AI functions, specifically differentiating ideational scaffolding from textual production, within the learner's Zone of Proximal Development.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.