Patterns of Developer Adoption of LLM-Generated Code Refactoring Suggestions
David Schön, Faiza Amjad, Tehreem Asif, Ranim Khojah, Mazen Mohamad + 2 more
TLDR
This paper analyzes how developers adopt LLM-generated code refactoring suggestions, finding most are accepted, with major changes following five patterns.
Key contributions
- Analyzed 169 GitHub commits where developers refactored code using ChatGPT suggestions.
- Found developers mostly accept LLM refactoring suggestions without modifications.
- Identified five distinct patterns for major modifications when developers do change suggestions.
- Modification patterns depend on refactoring activity, developer prompt, and ChatGPT response validity.
Why it matters
Current research evaluates LLM refactoring suggestions, but practical developer adoption is less understood. This study fills that gap by analyzing 169 GitHub commits, revealing how developers use and modify LLM-generated code. Understanding these patterns can inform the design of more effective LLM tools and developer workflows.
Original Abstract
Large language models (LLMs) have gained widespread popularity and have steadily improved over time, enabling software developers to use them for various code-related tasks. One common task is code refactoring, where the LLM suggests changes for the developer to apply to their code to improve quality attributes such as readability or maintainability. While current research focuses on evaluating LLM-generated refactoring suggestions, there is a limited understanding of how developers apply these suggestions in practice. To explore this, we analyze 169 GitHub commits where developers refactor their code based on a ChatGPT conversation linked in the commit message. We found that developers mostly accept and use the suggestions without modifications. When changes are made, they are mostly major and fall into five different patterns that depend on the refactoring activity, the developer's prompt, and the validity of the response from ChatGPT.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.