InfoWorld's Eval Hygiene Piece Is the Most Honest Postmortem the AI Industry Has Published All Year

The most useful thing Anthropic has published in months is not a model. It is a postmortem.

In March and April of this year, Anthropic shipped three regressions to Claude Code users in rapid succession. None of them were caught by their internal eval pipeline. One cost 3% on coding quality. One introduced a latency flip that swapped reasoning effort for speed based on a misread signal. One was a caching bug that cleared thinking state on every turn instead of after one idle hour. Users noticed all three before the eval systems did. Anthropic then wrote up what went wrong, published it, and let the community read it. That sequence — mistake, postmortem, publication — is more instructive than any benchmark the company has ever released.

InfoWorld's May 3 piece anchors its argument in this postmortem and arrives at a conclusion that the industry has been circling for a year without quite landing: the AI code quality problem is not fundamentally a model problem. It is an eval infrastructure problem. And eval infrastructure is something most teams are treating as an afterthought.

The three failure modes in the Anthropic postmortem map to three distinct eval weaknesses, and each one is worth sitting with.

The first was a latency flip. The engineering team saw that switching default reasoning effort from high to medium produced "slightly lower intelligence with significantly less latency." They shipped it. The problem is that "slightly lower intelligence" is not a number. It is a vibe. The eval that approved this change was measuring latency — a clean, countable metric — and not measuring the qualitative coding degradation that came with it. The lesson is not that Anthropic's engineers are careless. It is that optimizing for a metric you can count while ignoring the metric you cannot is the natural gravitational pull of any measurement system, and AI quality is mostly the second kind.

The second was a caching optimization. The team intended to clear stale thinking after one idle hour. A bug shipped that cleared it on every turn. The eval system did not catch this because the change was treated as a performance optimization rather than a behavioral change. Performance optimizations and behavioral changes are different kinds of diffs, but the agent loop does not always know which kind it is touching. This is a specific instance of a general problem: side-effect surface area in agent systems is larger than most teams are accounting for, and eval coverage assumptions tend to lag behind the actual blast radius of changes.

The third was a prompt change — two lines asking Claude to be more concise. It passed the standard release gate. It failed an extended ablation suite that the team ran later and only deployed after the regression was already in users' hands. The standard gate was too narrow to catch what "more concise" does to a coding agent's output quality when concision trades against completeness in complex refactoring tasks.

Together these three cases describe a pipeline that looks solid until you stress it with the right kind of breadth. Most teams have an eval suite. Few teams have an eval suite that covers behavioral side effects, optimization-turned-regression paths, and prompt sensitivity across diverse task types simultaneously. That is not a criticism — building that kind of coverage is genuinely hard. But it is an honest description of where the gap lives.

The piece brings in Angie Jones's perspective from the Agentic AI Foundation with an observation that cuts through the hype layer: "a lot of the problems people blame on AI are actually problems that always existed, AI just amplified them." This matters because it redirects the conversation from "AI is unreliable" to "our reliability infrastructure was calibrated for a different failure mode." Code review missed these regressions too. Internal users missed them. The eval pipeline missed them. That is not a story about AI being broken. It is a story about the feedback loops that worked for traditional software engineering being underpowered for the kind of continuous behavioral drift that agent systems introduce.

The piece also surfaces Karpathy's autoresearch experiment — 700 runs over two days with binary keep-or-revert decisions — as a structural model for how eval-driven development could work at scale. The experiment is described rather than analyzed in depth, but the implication is clear: the way to make vibe coding survivable is to replace vibes with actual outcome measurement, and to do it at a volume that matches the iteration speed of the systems you are building.

LangChain's April update of 30+ evaluator templates gets a mention as the kind of scaffold that makes this tractable for teams who are not Anthropic. That is a fair point, but it undersells the organizational dimension. Eval templates are available. Eval discipline is a product decision. Teams that treat eval infrastructure as something a developer sets up once and then ignores will have the same blind spots as teams that treat monitoring as a dashboard deployment rather than an on-call commitment. The teams that get this right are usually the ones where someone with authority decided that eval debt is technical debt and budgeted accordingly.

There is one more point in the piece worth extracting and sitting with separately: the distinction between pass@k and pass^k. A system that succeeds 75% of the time sounds acceptable in isolation. Run it three consecutive times — which is what you do when an agent handles a multi-step task and each step could fail independently — and you are down to 42%. That is not a 75% reliable system. That is a system that fails more often than it works across realistic end-to-end scenarios. The piece mentions this but does not belabor it, which is appropriate. The implication for practitioners who are building customer-facing agent workflows is that pass@1 is the only metric that actually describes the experience your users will have. Most teams are measuring something else without realizing it.

What should engineers do with this? The piece's recommendations are sound — write evals before prompts, make regression testing a release gate, treat user complaints as eval inputs — but they land better when you understand the underlying dynamic. Eval infrastructure is not a box you check at project start and then consider handled. It is a living product that needs the same maintenance, ownership, and iteration budget as the agent system it is guarding. The teams that ship reliable AI code in 2026 will not be the ones with the best models. They will be the ones with the most honest feedback loops, and feedback loops are made of eval infrastructure, not good intentions.

Sources: InfoWorld, Anthropic Engineering, Andrej Karpathy