AI Benchmarks Have a Reproducibility Problem, and Evaluation Cards Finally Names the Missing Fields

Benchmark leaderboards have become the model industry’s favorite kind of evidence: simple enough to screenshot, ambiguous enough to survive scrutiny. Evaluation Cards, a new arXiv paper and live reporting layer, is useful because it says the quiet part out loud. The problem is not that AI evaluations are impossible. The problem is that the industry keeps publishing scores without enough receipts for anyone else to understand what was actually measured.

The paper, authored by a large multi-institution group including Avijit Ghosh, Anka Reuel, Yacine Jernite, Stella Biderman, Sanmi Koyejo, Irene Solaiman, and many others, proposes an interpretive layer that joins benchmark metadata, model metadata, and individual evaluation-run data into one record. That sounds like documentation plumbing. It is. It is also exactly the plumbing missing from most model claims.

The audit behind the proposal is the important bit. The authors report applying Evaluation Cards across 5,816 models, 635 benchmarks, and roughly 102,000 reported results from 30 organizations. Their schema comes from a structured review of 52 papers and 10 stakeholder interviews, then renders four signals: reproducibility, documentation completeness, provenance and risk, and score comparability. In other words: can someone rerun this, can someone understand the benchmark, who made the claim, and is this score even comparable to the other score sitting next to it?

The leaderboard economy has been running without commit metadata

The finding that should make practitioners pause is brutal: 96.5% of (model, benchmark, metric-path) triples lack at least one minimal reproducibility field. Missingness is especially high for max_tokens at 95.6% and temperature at 93.9%. For agentic benchmarks, fields like eval_plan and eval_limits are missing for 100% of agentic benchmark triples.

Those are not clerical details. Temperature, token limit, prompt format, tool budget, retry policy, benchmark split, and scorer variant can all change the result. A vague “MATH score” or “SWE-bench score” without the rollout hierarchy — family, composite, benchmark, split, metric — is like publishing a database benchmark without hardware, cache state, query plan, or dataset size. It may be directionally interesting. It is not deployment evidence.

The paper also reports that median per-benchmark documentation completeness is only 10.7% across the 635 benchmarks. First-party reporting is worse than third-party reporting in paired cases: developer self-reports populate 0.0% of the base reproducibility fields on average, compared with 16.6% for third-party reports. That does not prove vendors are cheating. It does prove that vendor score cards are often optimized for announcement velocity, not downstream engineering decisions.

There is another uncomfortable number: 98.2% of model-benchmark pairs are reported by only one party. Among multi-organization metric groups, 94 of 181 — 51.9% — have cross-party divergence flags. In plain English, most results never get a second measurement, and when they do, disagreement is common. That is a weak foundation for how much money companies are now spending on model procurement, routing, and agent infrastructure.

Agent evaluations are where this gets expensive

Evaluation Cards is especially timely because agent benchmarks are replacing classic chat benchmarks as the prestige surface. That shift makes missing metadata more dangerous, not less. An agent run depends on the model, yes, but also on the harness: shell access, browser behavior, file tools, timeout rules, memory, subagent parallelism, hidden retries, stop conditions, and whether the agent is allowed to inspect tests before editing code. Change the harness and the “model score” can become a runtime score in disguise.

For engineering teams, the takeaway is not “ignore benchmarks.” That would be lazy. Benchmarks are useful when they are treated like tests, not advertisements. The right move is to demand cards for any score used in a decision: exact model version, prompt template, sampling settings, context budget, tool permissions, eval limits, scorer, task split, run date, reporter identity, and examples of failures. If those fields are absent, mark the claim as non-comparable until proven otherwise.

This should also change internal evaluation practice. Teams comparing GPT, Claude, Gemini, Qwen, or local models should create their own internal evaluation cards. Record the setup every time a model is tested against a customer-support workflow, code review harness, data-analysis agent, or compliance extraction task. Store not just aggregate pass rates but the task-level failures, correction rate, cost per successful task, latency, retry count, and escalation rate to a stronger model. If you cannot reconstruct why a model won last month, you cannot safely route production work to it this month.

One original implication here is procurement discipline. Enterprise buyers should stop asking vendors for “your benchmark numbers” and start asking for “your evaluation card for this benchmark.” A vendor that publishes incomplete first-party scores may still have a strong model, but the buyer should price in replication work. The missing metadata becomes a tax: somebody on your team will have to recreate the conditions, discover the hidden assumptions, and find out whether the improvement survives your harness.

Another implication is leaderboard design. Public leaderboards should stop displaying naked aggregate scores as the default view. The default should include comparability warnings, reproducibility completeness, and provenance labels. A lower score with a complete, independently reproducible card can be more decision-useful than a higher score with no setup details. That is not anti-progress. That is how engineering works when the stakes move beyond demo week.

The live Evaluation Cards site matters because standards only help if they enter the workflow. Model cards and system cards became useful once people expected them; evaluation cards need the same pressure. The industry does not need fewer benchmarks. It needs fewer benchmark numbers pretending to be stronger evidence than they are.

My read: Evaluation Cards is the kind of boring infrastructure paper that should become embarrassing not to comply with. If a model claim cannot tell you how the number was produced, it is not a result yet. It is a pull request with a green badge and no test log attached.

Sources: arXiv: An Interpretive Layer for AI Evaluation Reporting, Evaluation Cards live reporting layer, CORE-Bench reproducibility context, SWE-bench leaderboard context