RigorBench Says Coding-Agent Leaderboards Are Measuring the Answer, Not the Engineering

Most coding-agent benchmarks still grade agents like contest submissions: did the patch pass, did the issue close, did the scoreboard increment? That is useful, but it is also a dangerously compressed view of software engineering. RigorBench, a new arXiv paper on “engineering process discipline” in autonomous AI coding agents, argues that the path to the patch matters — and anyone who has reviewed an agent-generated diff that somehow passed tests while detonating half the codebase should be nodding already.

The paper’s basic complaint is simple: outcome-only benchmarks reward the agent that eventually lands on green, even if it got there by thrashing, ignoring its own plan, skipping verification, burning tokens in a doom loop, or leaving the repository in incoherent intermediate states. That is not a small methodological nit. It is the difference between an assistant you can put in a production workflow and a stochastic intern with shell access.

Green tests are necessary. They are not a trust model.

RigorBench evaluates autonomous coding harnesses across five dimensions: Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity. The benchmark contains 30 tasks grouped into five categories with names that read like they came from a senior engineer’s review checklist: Plan-Then-Build, Verify-Or-Die, Doom Loop Gauntlet, Know When to Fold, and Don’t Break the Build.

That framing is the important contribution. A coding agent’s final diff is only one artifact. The full trajectory includes its reasoning text, file edits, command executions, test outputs, timestamps, token consumption, and action ordering. RigorBench instruments those trajectories inside isolated Docker containers, with fresh task repo clones, a 60-minute timeout, and a 200K-token context budget. The paper compares Agent-Rigor, Agent-Skills, Superpowers, and a baseline ReAct loop across 120 task executions.

The reported result is that structured process discipline improves process quality by an average of 41% and downstream outcome correctness by 17%. The headline scores put Agent-Rigor at 0.61 RigorScore and 0.83 Outcome, Agent-Skills at 0.47 and 0.72, Superpowers at 0.48 and 0.70, and the baseline ReAct loop at 0.48 and 0.64. There is a caveat: the brief notes some inconsistency between score tables and prose, so the exact numbers should be treated with the same scrutiny the paper asks us to apply to agents. Still, the direction of travel is right.

Software teams already evaluate this way informally. When a human engineer opens a pull request, reviewers do not only ask whether CI passed. They ask whether the change is minimal, whether the test proves the bug, whether the migration path is safe, whether failure cases were considered, whether the author understood the system, and whether the patch can be rolled back. Coding-agent evals have been pretending that all of that collapses neatly into one pass/fail bit. It does not.

The process is part of the product

The strongest dimension in RigorBench may be Abstention Quality. Agents are usually marketed as tireless problem solvers, but a competent engineering agent should sometimes refuse to proceed. If requirements are contradictory, the repository state is inconsistent, a critical dependency is unavailable, or the requested change would require production credentials, “I cannot safely complete this” is not a failure. It is an essential capability.

Recovery Efficiency is equally practical. Long-running coding agents often fail not because the first hypothesis is wrong, but because the second through tenth hypotheses are just increasingly expensive variations of the same mistake. A benchmark that penalizes doom loops forces vendors to optimize for diagnosis, rollback, and plan revision instead of raw persistence. Persistence looks great in demos. In a real repo, it becomes CI spam, review fatigue, and a very large token invoice.

Atomic Transition Integrity is the boring systems criterion that deserves more attention. Agents need to keep the codebase coherent between steps. Half-applied refactors, unrelated formatting churn, dead files, abandoned migrations, and test fixtures mutated to satisfy the patch are all process failures even when the final result can be massaged into passing. This is where agent benchmarks should start borrowing from database thinking: intermediate states matter when another tool, reviewer, or agent may observe them.

For practitioners, the takeaway is not “replace SWE-bench.” SWE-bench, LiveCodeBench, Terminal-Bench, and similar benchmarks still answer useful questions about task success. The move is to add trajectory evaluation on top. If you are trialing Claude Code, Codex, Cursor, Qwen Code, OpenCode, or an internal harness, log plans, edits, commands, tests, retries, rollbacks, stop reasons, and token use. Then score successful tasks for whether the path would pass code review, and score failed tasks for whether the agent failed safely.

That last phrase matters. A failed agent that stops with a clean diagnosis and no destructive edits may be more valuable than a successful-looking agent that reaches green by deleting coverage, relaxing assertions, or silently changing public behavior. Outcome-only leaderboards blur that distinction. Engineering organizations cannot afford to.

Benchmarks need audit trails too

RigorBench uses deterministic heuristics plus an LLM-as-judge panel of three models with majority voting for subjective process scoring. That is reasonable for early research, but it creates the next review item: benchmark artifacts must be inspectable. If process discipline becomes a number on a vendor slide, teams need access to rubrics, trajectories, judge prompts, disagreement rates, and examples of borderline calls. Otherwise we have merely replaced one opaque scalar with another.

There is also a risk that agents will learn to perform “good process” theatrically. A model can write a plan without following it. It can run tests without choosing the right tests. It can apologize, revise, and produce plausible self-critique while still making the wrong patch. Process benchmarks need to inspect actions, not just prose. The shell history and diff graph are more trustworthy than the agent’s narration of its own competence.

The useful future is a blended scorecard: final correctness, process discipline, cost, latency, review burden, and blast radius. That is how teams actually choose tools. The best coding agent for a codebase is not necessarily the one with the highest public pass rate; it is the one that produces understandable changes, verifies them cheaply, recovers from mistakes, and leaves reviewers with less work rather than a more interesting class of cleanup.

RigorBench is not the final word. It is a needed correction. Coding agents are moving from toy tasks into real development loops, and real development is not just answer generation. It is disciplined change under constraints. If the benchmark cannot see the discipline, it is not measuring the product engineers are buying.

Sources: arXiv, SWE-bench, Terminal-Bench, LiveCodeBench