VaSE Makes Reasoning Models Cheaper by Protecting the Tokens That Matter

The next cost fight for reasoning models is not just token price. It is memory. Every long “thinking” trace leaves behind key/value cache entries the decoder has to keep using, and that cache becomes the hidden tax on local inference, high-concurrency serving, and agent workflows that generate thousands of tokens while debugging their own mistakes. VaSE is interesting because it asks a very practical question: which cached thoughts are safe to forget?

The paper, Value-Aware Stochastic KV Cache Eviction for Efficient Long-Range Reasoning, proposes a training-free cache eviction method for reasoning models. The authors evaluate it on Qwen3-4B and Qwen3-14B across six reasoning tasks spanning math, code generation, and science QA. Unlike approaches that require retraining or model-specific architectural changes, VaSE sits in the inference path. That makes it immediately relevant to teams trying to serve smaller reasoning models locally or privately without turning GPU memory into the limiting reagent.

The core observation is specific enough to be useful: some value vectors in the KV cache have unusually large magnitudes, and evicting them can cause reasoning models to collapse into repetitive loops or malformed output. That is a sharper diagnosis than the usual “long context is expensive” hand-wave. It says the cache is not just a chronological transcript. Some entries carry disproportionate downstream influence, even when a simpler eviction policy might consider them expendable.

Not all cached tokens are equal

Most engineers encountering KV-cache optimization think first about recency, attention scores, redundancy, or block sparsity. VaSE puts the value vectors themselves under the microscope. The paper reports that deliberately evicting large-magnitude value states drops Qwen3-4B GSM8K accuracy to 14.3%, underperforming random eviction by 38.9% at a 512-token budget. That is not a small degradation. It is the model losing the plot because the eviction policy removed states that mattered more than their position in the sequence suggested.

VaSE combines two signals: protect large-magnitude value states, and add stochasticity so the retained cache stays diverse instead of collapsing into a brittle deterministic subset. On GSM8K, value scoring boosts Qwen3-4B accuracy by as much as 16.2%, while stochasticity adds another 4.7%. The engineering moral is familiar but easy to forget: optimization policies need to preserve both importance and coverage. Keep only the obvious tokens and you may overfit the cache to the model’s current path; keep random tokens and you throw away load-bearing state.

At roughly 4× KV-cache compression — about 25% of full KV activated — VaSE-AttnV averages 59.09% on Qwen3-4B, slightly above SeerAttention-R at 58.81% and above R-KV at 54.69% by 4.4 points. On Qwen3-14B, VaSE-AttnV averages 65.81%, again narrowly above SeerAttention-R at 65.37% and above R-KV at 60.90% by 4.9 points. VaSE-DKV improves over CurDKV by 7.7 points on Qwen3-4B and 9.2 points on Qwen3-14B.

The comparison to selection methods matters. Sparse selection can reduce compute by activating a subset of cached entries, but if the full cache is still retained, memory continues to grow with sequence length. Eviction caps memory. That is the product distinction. If you are running a local agent on a workstation, serving multiple tenants on a GPU, or trying to keep a reasoning model alive for long outputs, static memory is not a nice-to-have. It is the difference between a successful run and an out-of-memory error wearing a lab coat.

The product angle is static memory, not benchmark glitter

The throughput benchmark gives the paper its operational bite. On a single A100-80G, using Qwen3-14B, 16K output tokens, and a KV-cache budget of 2048, VaSE-DKV reaches 411 tokens per second versus 133 tokens per second for full cache — a 3.1× speedup. The benchmark sweeps KV budgets of 2048, 4096, and 6144 and output lengths of 16,384 and 32,768, using FlashAttention2 kernels without PagedAttention, batch size 16, and input prompt length 256. The full-cache Qwen3-14B setting is reported as OOM at 32K output, while eviction methods stay within the static budget.

Those details matter because inference papers are easy to misread. A 3.1× speedup on A100 with FlashAttention2 and no PagedAttention does not mean your vLLM deployment will automatically get the same number. It does mean teams should stop evaluating reasoning models only on answer quality and first-token latency. Long-output throughput, peak memory, OOM rate, retry behavior, and degradation under cache pressure are now first-class product metrics.

This connects directly to coding-agent economics. A coding agent does not just answer one question. It reads files, drafts a plan, edits code, runs tests, interprets logs, retries, explains itself, and often carries a transcript much larger than the original prompt. If each long reasoning run increases GPU memory pressure until concurrency collapses, your “cheap local model” becomes expensive in a different currency: throughput, reliability, and engineer patience. VaSE is the kind of inference plumbing that can make a 14B reasoning model less fragile in those loops.

It also pairs naturally with the broader local-agent story. NVIDIA’s DGX Spark and Qwen-focused runtime work make the hardware and serving stack more accessible. Quantization reduces model footprint. PagedAttention and FlashAttention improve serving mechanics. Cache eviction methods like VaSE attack the long-reasoning tail, where memory grows because the model keeps thinking. None of these pieces alone makes local agents production-ready. Together, they turn “run a useful model privately” from a weekend project into something closer to infrastructure.

There are caveats. The paper evaluates Qwen3-4B and Qwen3-14B; value-state behavior may differ across MoE models, multimodal systems, or closed frontier models. Stochastic eviction also raises reproducibility questions. If your agent workflow relies on deterministic replay for audits or incident review, you need to measure whether stochastic cache behavior changes outputs in unacceptable ways. And because the benchmark uses specific kernels and serving assumptions, production teams should test against their actual stack rather than importing the headline speedup.

The practitioner move is straightforward. If you serve reasoning models, add KV-cache policy to your evaluation matrix. Compare full cache, sparse selection, eviction, quantized cache, and your production serving defaults on real agent traces, not toy prompts. Measure answer quality, loop frequency, malformed output, memory, throughput, OOMs, and cost per completed task. The failure mode VaSE identifies — repetitive loops after evicting the wrong states — is exactly the kind of bug a generic tokens-per-second benchmark will miss.

VaSE is not glamorous in the way a new frontier model is glamorous. That is the point. The AI stack is moving from “can the model reason?” to “can we afford to let it reason for long enough, often enough, without the serving layer falling over?” The next generation of agent cost control will not come only from smarter routing or cheaper APIs. It will come from knowing which cached thoughts are worth keeping — and which ones can be safely forgotten.

Sources: arXiv, arXiv HTML, VaSE GitHub, FlashAttention, SeerAttention, Qwen