CodeChat-Eval Measures the Thing Copilots Actually Do: Break Code on Turn Seven
The most honest coding-assistant benchmark is not the one where a model writes a function and leaves. It is the one where the user comes back nine times asking for small refinements until the code breaks in a way that looks reasonable in the chat and wrong in the test suite.
CodeChat-Eval is useful because it evaluates that failure mode directly. Instead of treating code generation as a one-shot event, the benchmark builds 10-turn sessions: one initial generation turn followed by nine refinement turns. The follow-ups include cosmetic, structural, and semantic instructions, with operations like add, modify, remove, and replace. The benchmark starts from 542 software-engineering tasks and evaluates eight models, including Llama 3.1 8B, Llama 3.3 70B, Qwen 2.5 Coder 7B/14B/32B, DeepSeek-V3, GPT-5 Nano, and GPT-5.
The result is the thing every developer has seen in miniature. Functional correctness declines across the dialogue for every evaluated model. Reported drops from turn 0 to turn 9 include Llama 3.1 8B at -69.20 percent, Llama 3.3 70B at -56.38 percent, Qwen 2.5 Coder 7B at -52.03 percent, Qwen 2.5 Coder 14B at -39.65 percent, Qwen 2.5 Coder 32B at -42.86 percent, GPT-5 Nano at -24.41 percent, and GPT-5 at -27.12 percent on one of the reported task sets. GPT-5 has the best Mean Sustainable Turns score, MST@10 = 6.894 across both datasets. GPT-5 Nano scores 6.423. Llama 3.1 8B scores 2.308.
Single-turn coding benchmarks still matter. They test whether a model can synthesize a correct solution from a prompt. But they understate the main product risk in copilots and coding agents: the model can maintain the conversation while losing the program. It can satisfy the latest instruction, preserve the style the user asked for, and still violate a behavior that was established four turns earlier.
The dangerous request is the harmless-looking one
CodeChat-Eval's finding that semantic instructions and add-style changes produce the largest regressions should surprise nobody who has used these tools seriously. “Add support for this edge case” sounds local. “Also handle lists” sounds obvious. “Make it cleaner” sounds cosmetic until the model turns a guard clause into a broad branch and changes behavior for null input. Real coding work is full of requests that look small in natural language but have wide semantic blast radius.
This is where chat UI design can make the problem worse. A conversation makes continuity feel natural. The model responds fluently, refers to the previous code, and formats the patch neatly. That surface continuity can mask behavioral drift. Developers are good at reviewing diffs, but a chat assistant often encourages reviewing intent: does this look like it did what I asked? The better question is: does this still do everything the earlier version did?
The benchmark uses EvalPlus as the execution harness to reduce false positives from weak tests, which is the right instinct. Multi-turn code evaluation without strong tests becomes a style benchmark wearing a compiler costume. The paper also reports manual verification numbers: 91 percent applicability-judgment accuracy with Cohen's Kappa 0.709, and 75 percent instruction-adherence judgment accuracy. That second number is a reminder that even deciding whether a refinement was followed can be fuzzy. Functional correctness gives the benchmark a harder anchor.
The SKIP rate is also worth noting. Only 0.015 percent of the 39,024 follow-up turns contained SKIP flags, so the Mean Sustainable Turns metric is not being propped up by models refusing to engage. The degradation is happening while models participate. That is the product danger: the assistant keeps helping until the code is wrong.
Regression testing is the real memory
The practitioner takeaway is not that models are useless at iterative coding. It is that memory in a coding assistant cannot live only in the prompt. The durable memory of software is the test suite, type system, static analysis, build graph, and review process. If the agent is asked to perform turn seven without re-running the behavior from turn one, it is doing software development without the software's immune system.
Teams building coding agents should make multi-turn regression a first-class eval. Track pass rate not just after the initial solution, but after each refinement. Measure whether failures cluster around semantic adds, structural rewrites, or removals. Keep the original tests active throughout the session. Add generated tests cautiously, but do not let generated tests replace the original contract. If a model edits code in a long-running conversation, require it to explain the preserved invariants and run the checks that prove them.
Product teams should also rethink how assistants present iterative changes. A patch after turn seven should surface what changed, what invariants were supposed to remain true, which tests were run, and which prior requirements might be affected. That is not extra ceremony. It is the difference between a coding assistant and a confident autocomplete loop with amnesia.
For engineering managers evaluating AI coding tools, CodeChat-Eval points to better procurement questions. Do not ask only how the tool scores on HumanEval, MBPP, or a vendor's internal benchmark. Ask how correctness changes over a multi-turn session. Ask whether the agent reruns prior tests after refinements. Ask whether it can preserve behavior while applying style changes. Ask how it handles contradictory follow-ups. Ask whether it identifies when a request is semantically risky instead of treating every instruction as a local edit.
There is a broader benchmark lesson here. The industry has spent years over-weighting first-answer quality because it is easy to measure and easy to market. But software engineering is not a first-answer task. It is an iterative control problem over a changing artifact. The model has to preserve intent, code behavior, user preferences, and test evidence across turns. A benchmark that asks whether the assistant breaks code on turn seven is closer to the daily experience than another leaderboard row for one-shot generation.
The take: copilots should be judged on sustainable correctness, not just initial brilliance. If your assistant can produce a good first draft but erodes behavior with every polite refinement, it is not ready to own a workflow. It is ready to sit next to a test runner and be watched carefully.