EvoArena Shows Why Agent Memory Needs Git-Style History, Not Just a Bigger Scratchpad

Agent memory keeps getting described as a bigger notebook. EvoArena makes the better argument: it should look more like git history.

The benchmark suite evaluates agents in environments that change over time: terminal workflows, software repositories, and user preferences. Its paired method, EvoMem, wraps ordinary memory systems with append-only patch histories that preserve what changed, what the previous state was, why the update happened, and which evidence justified it. This sounds boring until you have watched an agent confidently apply last week's deployment command to this week's infrastructure and break the one thing the migration was supposed to protect.

Most agent benchmarks are static snapshots. That is convenient for scoring, and increasingly misleading for production. Real environments drift. CLIs change flags. APIs deprecate fields. Repos move modules. Tests get stricter. Permissions move. Users revise preferences. A useful agent has to know not only the latest rule, but also when the old rule still applies, why the rule changed, and what evidence supports the current version. A summary blob called “memory” is not enough.

EvoArena turns that deployment problem into an evaluation target. It includes Terminal-Bench-Evo for evolving terminal workflows, SWE-Chain-Evo for evolving codebases, and PersonaMem-Evo for evolving user preferences. The paper reports that current agents average 39.6% accuracy across these evolving domains. That number should not surprise anyone running agents against real systems. Static benchmarks reward solving the task in front of you. Production rewards solving the task without forgetting the history that made the current state true.

State collapse is the quiet agent failure

The paper names a failure mode that deserves to become standard vocabulary: state collapse. A memory system stores “the current truth,” overwrites the old truth, and loses the reason the old truth existed. That works when new information safely supersedes old information. It fails when reality is versioned. An old command may still apply to an older branch. A preference may be conditional. A policy may have changed for one workspace but not another. A repo migration may be halfway complete.

EvoMem's patch model is the obvious fix in hindsight. Each memory update stores the pre-update memory, the post-update memory, the update rationale, and supporting evidence. At inference time, the agent retrieves from the latest memory by default but can pull relevant patches when a query depends on overwritten states, conflicting evidence, or earlier versions. That is exactly how engineers already work. We do not just keep the current file; we keep commits, diffs, issue links, tests, migration notes, and blame. Memory without history is a production incident waiting for a prompt.

The results are modest in average percentage terms and more interesting in shape. EvoMem improves EvoArena by 1.5% on average, GAIA by 6.1%, LoCoMo by 4.8%, and chain-level accuracy on EvoArena by 3.7%. The project page gives more granular signals: Terminal-Bench-Evo average step accuracy improves from 43.6 to 46.0, while chain accuracy moves from 21.5 to 27.6. SWE-Chain-Evo step accuracy barely moves, 27.9 to 28.3, but chain accuracy improves from 10.0 to 12.1. PersonaMem-Evo step accuracy goes from 47.3 to 49.0, with chain accuracy from 40.0 to 43.2.

The chain numbers are the part to stare at. Step success can make an agent look competent while the overall workflow remains broken. That is how demos lie. A coding agent can fix the current issue and break a previous invariant. An ops agent can complete a command sequence and lose the compatibility requirement. A personalization agent can remember the latest preference and forget the exception. Chain accuracy measures the thing users actually experience: did the agent preserve coherence across time?

What this means for coding and ops agents

If you run coding agents, incident agents, or workflow automations, the immediate takeaway is to instrument memory patches. Record dependency upgrades, command migrations, API version changes, policy updates, user preference revisions, repo conventions, and validation failures as structured deltas, not overwritten summaries. The patch should say what changed, why it changed, where the evidence came from, what still applies, and which validator confirmed it. If that sounds like extra work, compare it with debugging an agent that silently learned the wrong current truth.

This also changes how teams should evaluate agents. Do not only measure individual task success. Measure sequences. Run regression-preservation tests after the agent updates memory. Ask whether it can handle rollback, branch-specific behavior, versioned APIs, and conflicting user preferences. For SWE-style agents, measure PASS_TO_PASS-style preservation, not just whether the current patch passes one issue's tests. EvoArena reports SWE-Chain-Evo PASS_TO_PASS failure rates dropping from 9.09% to 6.32% with EvoMem. That is the kind of reliability metric production teams should care about.

There is a product lesson too. Bigger context windows are not a substitute for version-aware memory. A million-token scratchpad can still be a junk drawer. The hard part is not storing more facts; it is representing causality and validity. When did this become true? What did it replace? Is the old version still valid somewhere? Which evidence supports the update? Which test failed before the change and passed after it? These are memory schema questions, not context-length questions.

The caveat: EvoArena's repository is being released progressively, and teams should check the state of the code, datasets, and license before building on it directly. But the benchmark's framing is already useful. Agents are moving from one-shot assistants to persistent operators. Persistent operators need historical state, not just recent chat.

The take is simple: reliable agents need memory that can survive time. If your memory system cannot show the diff, it probably cannot explain the decision. And if it cannot explain the decision, it is only a matter of time before it applies the right rule to the wrong version.

Sources: arXiv, EvoArena project page, GitHub repository, Hugging Face collection