TokenPilot Says the Expensive Part of Long-Running Agents Is Not Just Tokens — It Is Cache Invalidation

The expensive part of a long-running agent is not always the tokens you can see. Sometimes it is the cache hit you accidentally destroyed three turns ago because your framework rewrote the top of the prompt, moved the tool schema, inserted a fresh timestamp, or “helpfully” summarized the wrong block. TokenPilot is useful because it says the quiet part out loud: agent cost control is no longer just a token-counting problem. It is a cache-continuity problem.

The new arXiv paper, Cache-Efficient Context Management for LLM Agents, proposes TokenPilot, a context-management framework for long-horizon LLM agents. Its premise is straightforward and under-discussed: existing pruning and memory-eviction systems often reduce text volume by mutating the sequence layout. That can break prompt-prefix matching, invalidate the provider’s KV cache, and force the serving stack to recompute tokens that should have been cheap.

That matters because modern model pricing increasingly distinguishes between fresh input and cached input. If the stable prefix of your agent prompt can be reused, repeated turns can be much cheaper. If your framework changes the prefix every time, the bill behaves like every turn is new work. Congratulations, you compressed the prompt and still paid full freight.

Shorter context can be more expensive if it breaks the prefix

TokenPilot has two main pieces. The first is Ingestion-Aware Compaction, which stabilizes prompt prefixes and filters open-world environmental noise at the ingestion gate. The second is Lifecycle-Aware Eviction, which removes context segments conservatively only after their residual task utility has expired. The paper explicitly models serving cost as cache-hit tokens plus cache-miss tokens, with cache hits discounted by a rate α much smaller than 1. That is the conceptual upgrade: the optimization target is not “fewest tokens,” but “fewest expensive tokens without losing task state.”

In experiments on PinchBench and Claw-Eval, TokenPilot reports cost reductions of 61% and 56% in isolated mode, and 61% and 87% in continuous mode, while maintaining competitive performance against prior systems. The agent backbone is GPT-5.4-mini. Baselines include LLMLingua-2, SelectiveContext, Keep-Last-N, Summary, LCM, Pichay, MemoBrain, AgentSwing, and MemOS. The estimator is instantiated with Qwen3.5-35B-A3B as a lightweight zero-shot validator, and the paper says its total operational cost across the continuous PinchBench stream is less than $0.03.

The implementation signal is also worth noting. TokenPilot has been integrated into LightMem2, which exposes operational commands such as /lightmem2 status, /lightmem2 report, /lightmem2 doctor, /lightmem2 visual, and mode switches for conservative, normal, and aggressive behavior. That does not make it production-proven. It does make the paper more concrete than a PDF that ends with “future work.”

The LightMem2 numbers are the kind builders should like because they include tradeoffs. In the README’s PinchBench table, LightMem2 shows $3.22 in isolated mode versus $8.31 for vanilla, with an overall score of 81.0 versus 80.5. In continuous PinchBench, it reports $2.79 versus $7.24, with a score of 81.3 versus 79.2. But in continuous Claw-Eval, the cost drop is much larger — $10.58 versus $81.52 — while the score is lower, 60.8 versus 63.4. That is exactly the shape of a real systems result: useful, not magical, and requiring workload-specific validation.

Agent history is an execution log, not a chat transcript

The paper’s conservative eviction design is the part many agent frameworks should steal first. Long-running agent context is not a friendly chat history. It is an execution log with dependencies: user constraints, file reads, command output, errors, tool schemas, partial plans, failed hypotheses, and decisions that should not be re-litigated. Delete the wrong line and the model may repeat work, hallucinate state, or confidently edit against a stale assumption.

That is why “summarize everything older than N turns” is such a blunt instrument. It may keep the user’s intent while throwing away the one shell error that explained why the previous attempt failed. It may preserve the broad plan while losing the exact path of the generated file. It may rewrite a stable prefix in the process, converting a memory feature into a cache-miss generator.

TokenPilot’s lifecycle framing — active, completed, evictable — is a better mental model. The system should ask whether a segment still has residual utility, not whether it is old. A tool result that completed a branch of work may be safe to offload as a batch. A user constraint from the first turn may remain live for the entire session. A compiler error from ten turns ago may be dead if the file changed, or crucial if the agent is about to reintroduce the same bug.

The hard part is that lifecycle estimation becomes another moving part. If the estimator misclassifies utility, you can lose context that still matters. If it is too conservative, you save less money. If it is too aggressive, you buy lower cost by increasing behavioral risk. That is why the Claw-Eval score drop matters. Do not copy the mode switch and call it done. Measure your own failure modes.

What teams should instrument now

The practical takeaway for teams running coding agents, research agents, or support agents is simple: stop reporting only input tokens. That metric is now too coarse. Track cache reads, cache misses, stable-prefix length, tool-schema churn, environment-output volume, memory insertions, summarization frequency, and whether compaction changes early prompt boundaries. Then report cost per completed task, not cost per turn.

Cost per turn rewards the wrong behavior. An agent that spends fewer tokens per step but breaks cache locality and takes more steps may be more expensive. An agent that keeps a longer stable scaffold but reuses cached input may be cheaper. A system that aggressively summarizes may look clean in logs while silently deleting the evidence needed to finish the task. The only number that matters is whether the workflow completed correctly for less money and less latency.

This also changes how teams should compare model providers. The sticker price for Claude, Codex, Gemini, Qwen, or a local stack is only one component. The harness decides whether you repeatedly pay the expensive uncached rate. A cheaper model with bad context hygiene can lose to a more expensive model inside a cache-aware runtime. A provider with strong prompt-caching economics can still be wasted by a framework that rewrites the prefix every turn.

The larger industry signal is that agent economics are moving from model selection into runtime architecture. Context windows got huge, and the obvious response was to stuff more into them. TokenPilot points in the opposite direction: keep the stable parts stable, filter noise before it lands, evict only when lifecycle says the dependency is dead, and optimize for backend serving economics instead of screenshot-friendly token counts.

That is less glamorous than a new model release. It is also closer to where production agent costs are going to be won or lost. The next useful agent platform will not just ask “how much context can I fit?” It will ask “which context must remain byte-stable so I do not pay for it again?” That is the adult version of memory management.

Sources: arXiv: TokenPilot / Cache-Efficient Context Management for LLM Agents, LightMem2 implementation, PinchBench