Evaluating LLMs Code Reasoning Under Real-World Context
TLDR
R2Eval is a new benchmark for evaluating LLM code reasoning using 135 real-world Python problems, handling complex data types for practical assessment.
Key contributions
- Existing LLM code benchmarks are simplistic, using basic types and failing real-world project complexity.
- Introduces R2Eval, a benchmark of 135 code reasoning problems from 10 widely used Python projects.
- R2Eval serializes compound and custom data types to preserve real-world data complexity.
- Provides a more realistic assessment of LLMs' practical generalizability in code reasoning.
Why it matters
Evaluating LLMs on code reasoning is critical, but current benchmarks fall short by oversimplifying real-world code. R2Eval addresses this by offering a more realistic assessment using complex data types from actual Python projects. This improves our ability to gauge LLMs' practical utility.
Original Abstract
Code reasoning tasks are increasingly crucial to evaluating large language models (LLMs). Yet most existing benchmarks rely on simplistic, LLM-generated snippets or human-written solutions to code challenges and often restrict inputs and outputs to primitive types, failing to reflect the structure and dependencies of real-world projects. These simplifications limit their ability to measure practical generalizability. We present R2Eval1, a benchmark of 135 code reasoning problems drawn from ten widely used Python projects. Unlike prior work, R2Eval serializes compound and custom types, preserving real-world data complexity and enabling a more realistic assessment of LLMs.
📬 Weekly AI Paper Digest
Get the top 10 AI/ML arXiv papers from the week — summarized, scored, and delivered to your inbox every Monday.