OneHarness Makes the Coding-Agent Benchmark Problem Look Like an Integration Problem

The weakest part of most “best AI coding agent” comparisons is not the model take. It is the harness. If Claude Code gets the system prompt as a real system instruction, Codex gets it as user text, OpenCode starts in a different effective working directory, and Cursor cannot resume the same run cleanly, the benchmark is not measuring agent quality. It is measuring adapter luck.

That is why OneHarness v0.2.425 is more interesting than its popularity metrics suggest. The project had zero stars and zero forks at research time, which means this is not a community-hype story. It is an infrastructure story: a small cross-agent runner trying to make Claude Code, Codex, OpenCode, Goose, Qwen Code, Crush, GitHub Copilot CLI, and Cursor answer through one non-interactive interface with one normalized JSON envelope.

The release was published June 16 at 23:27 UTC with four assets. The README positions the tool as a single runner for multiple coding-agent CLIs, including parallel execution and a stable output shape. That output shape is the important part. OneHarness normalizes status, exit code, duration, command argv, final text, text source, usage, usage source, session ID, failure kind, stdout, stderr, and error. Not glamorous. Extremely necessary.

Benchmarks are integration tests wearing a leaderboard costume

The release notes and linked PRs read like a catalog of practical benchmark foot-guns. PR #12 fixed a particularly nasty one: the --system path previously reached Claude Code, but equivalent instructions did not reliably land in Codex, OpenCode, Qwen, Crush, Copilot, Cursor, and Goose. That is not a minor adapter bug. If one agent receives the evaluation rubric as privileged instruction and another receives it differently or not at all, the run is already invalid.

PR #3 fixed another boring-but-decisive mismatch: $PWD could diverge from --cwd because changing the child process directory did not update the inherited environment. OpenCode’s gate behavior diverged as a result. Anyone who has watched an agent claim “tests passed” from the wrong directory should feel the shape of that bug immediately. Working directory is not metadata; it is part of the task.

PR #6 added normalized usage, session, and failure signals across adapters. PR #87 added a layered config system: user-level ~/.config/oneharness/config.toml, project-level oneharness.toml or .oneharness.toml, CLI flag precedence, per-harness overrides, loud exit-2 errors for unknown fields or IDs, and a oneharness config debug view. That is the kind of plumbing serious evaluation requires before anyone should trust the chart at the end.

The useful lesson is that coding agents are no longer just models behind a prompt. They are runtimes with CLI flags, session IDs, context semantics, auth paths, tool permissions, output formats, and failure modes. Comparing them without normalizing those surfaces is like benchmarking databases while letting each one choose its own schema, isolation level, and cache warmup.

The cost story depends on the same boring fields

OneHarness also lands in the middle of the agentic cost-control conversation. GitHub’s AI Credits, Anthropic’s usage buckets, OpenAI’s token views, and provider-router projects all point to the same reality: teams need to know what an agent run consumed, whether it resumed cleanly, and how it failed. “This agent is cheaper” is meaningless if one harness drops usage on failure, another reports partial usage, and a third hides the final answer inside vendor-specific event logs.

Stable JSON is therefore not a developer-experience flourish. It is the audit surface. Platform teams need raw stdout and stderr for debugging, normalized usage for cost analysis, session IDs for reproducibility, and failure classifications for routing. A tool that cannot expose those fields may still be useful interactively, but it is weaker as managed infrastructure.

This matters for engineering leaders evaluating “Claude Code vs Codex vs OpenCode” more than it matters for people writing SEO comparison posts. A team deciding where to route repo-wide refactors needs to distinguish model failure from harness failure. Did the agent misunderstand the task, or did the safety prompt never land? Did it refuse correctly, or did the cwd mismatch trip a gate? Did it burn tokens on a runaway loop, or did the runner lose the usage report? Without the boring fields, every postmortem becomes vibes with logs attached.

What practitioners should do before trusting an agent bake-off

The practical checklist is short and uncomfortable. First, pin every CLI and harness version. Agent CLIs move quickly, and OneHarness explicitly had to modernize Codex flags after newer Codex builds removed -a never. Second, run an instruction-delivery fixture before running task-quality tests. Verify that system prompts, policy files, and repo conventions land in each harness with the intended authority.

Third, test cwd and environment semantics with a write-gated fixture. Have the agent report pwd, $PWD, the repo root, and a controlled file path before allowing it to mutate anything meaningful. Fourth, capture raw outputs alongside normalized fields. Normalization is useful; raw evidence is what saves you when the adapter is wrong. Fifth, treat missing usage, missing session IDs, or vague failure types as product limitations. Observability is part of the tool.

The bigger point is that the “best AI coding agent” question is not answerable at the model layer alone. The winning agent for a team will be the one whose runtime can preserve instructions, resume work, expose costs, respect workspace state, and fail in ways humans can diagnose. OneHarness is early, unproven, and not yet a community standard. But it is aiming at the right problem: the evaluation layer has to become engineering, not content marketing.

Sources: OneHarness v0.2.425 release, oneharness repository, PR #87, PR #6, PR #12