AGENTS.md Works When It Is Debugged Like Code, Not When It Is Treated Like Lore

AGENTS.md has become the new README-for-robots: part documentation, part incantation, part apology to the coding agent that is about to discover your monorepo’s build system. The useful question is no longer whether repository guidance helps. Sometimes it does. The question is whether teams are debugging that guidance like code, or treating it like lore.

Probe-and-Refine Tuning of Repository Guidance for Coding Agents gives the non-hand-wavy answer. The paper tests whether repository guidance files can improve coding agents on SWE-bench Verified, then shows the interesting bit: instructions work best when they are generated, tested, and iteratively refined against diagnostic probes. Static context is not magic. Evaluated context is a control surface.

The authors start from the increasingly familiar AGENTS.md pattern, but do not stop at “write some helpful notes.” They create synthetic bug-fix probes, use those probes to expose where guidance fails, and patch the guidance without running a full expensive agent loop during tuning. The result is less romantic than “agents understand your repo now,” and much more actionable: repository instructions can improve coverage, but they are model-specific artifacts that need evals, diffs, and deletion when they stop earning their token budget.

The gain is coverage, not sudden genius

The main experiment uses 500 SWE-bench Verified instances, four independent trials, and a 200-step agent budget in the principal Qwen condition. The model is Qwen3.5-35B-A3B with an effective 16k-token context and aggressive output truncation. Probe-and-refine guidance reaches a 33.0% mean resolve rate, compared with 28.3% for the static knowledge base used to initialize it and 25.5% for no context. The reported contrasts are statistically significant at p < 0.001 versus both baselines.

Those numbers are useful, but the mechanism is better. Refined guidance produces evaluable patches for 56.2% of instances versus 41.7% for no context, a 14.5 percentage-point coverage gap. Per-patch precision stays roughly constant at around 59%. In plain English: the guidance does not make the model much more correct once it has a patch. It helps the agent get to the point where a patch exists, applies, and can be judged.

That tracks with how repo instructions feel in practice. Good guidance rarely teaches a frontier model what a mutex is. It tells the agent which test command is real, which generated files to avoid, where the integration layer lives, which package manager has the lockfile that matters, and which architectural names are historical accidents. It reduces repository-specific thrash. That is valuable, especially under a step budget.

The step-budget result reinforces the same point. Guidance lets the agent use larger budgets productively; the unguided baseline stays comparatively flat while probe-refined guidance improves up to the 200-step condition. A longer leash helps only if the agent knows where to walk. Otherwise, it just spends more time inspecting irrelevant files with the confidence of a tourist holding the subway map upside down.

Portable skill packs should come with warning labels

The cross-model results are the part every agent platform should print out and tape to the roadmap. With NVIDIA-Nemotron-3-Nano-30B-A3B, no-context scores 28.4%, static guidance drops to 24.6%, and probe-refined guidance partially recovers to 27.0%. Worse, Qwen-tuned guidance applied to Nemotron collapses to 13.2%, compared with 27.0% for Nemotron self-tuned guidance and 28.4% for no context.

That is the paper’s sharpest practical lesson. Repository guidance is behavioral steering, not neutral documentation. A line that makes Qwen commit to a productive search path may make Nemotron overfit, hesitate, or wander. This should make teams skeptical of universal “agent skill packs” that claim to transfer cleanly across models and harnesses. Maybe some do. Prove it. The default assumption should be that guidance is tied to a model family, context budget, tool interface, and agent loop.

For engineering teams, the action item is straightforward: put AGENTS.md under CI-shaped discipline. Create a small suite of representative probe issues. Run the agent with no guidance, current guidance, and candidate guidance. Measure resolve rate, patch apply rate, test pass rate, steps, token spend, and failure modes. When an instruction does not change behavior, remove it. When it changes behavior for one model and hurts another, fork or condition it. Treat guidance as an executable artifact, not a documentation shrine.

This also changes how teams should write the file. Do not dump architecture essays into context because they feel important. Add operational facts the agent can use: commands, directories, invariants, forbidden edits, common traps, and decision rules. Keep it short enough that it survives the context budget. Include examples only when they reduce ambiguity. The best AGENTS.md is not comprehensive; it is selectively useful under pressure.

The broader editorial read is that context engineering is leaving the folklore phase. That is healthy. The industry spent a year discovering that models respond to instructions. The next year should be about measuring which instructions help, for which models, on which tasks, at what cost. AGENTS.md works when it is debugged like code. Otherwise it is just prompt folklore with a filename.

Sources: arXiv, project code, SWE-bench Verified, Qwen3.5-35B-A3B, NVIDIA Nemotron