4-bit KV Cache Is the Agent Cost Story Hiding Under the Model Benchmark
The most interesting AI-model cost story today is not a new leaderboard position. It is the thing that happens after your agent has read half a repository, opened a dozen tools, accumulated a giant prefix, and now needs to answer one tiny follow-up without turning the GPU into a space heater.
UltraQuant, a new arXiv paper on 4-bit key-value cache compression for context-heavy agents, is about that unglamorous but increasingly decisive layer. The authors are not arguing that FP4 cache is magically free. They are arguing something more useful: for multi-round agent workloads, the serving optimum changes once cache residency, concurrency, and time-to-first-token become first-class constraints.
That distinction matters because agent economics have been badly flattened into token-price tables. Teams compare Claude, Codex, Gemini, Qwen, DeepSeek, or local models by input and output cost, then discover the actual product bottleneck is not the nominal token rate. It is whether long prefixes can stay resident while thousands of short agent turns interleave on shared inference hardware.
The late turn is where the bill shows up
UltraQuant targets exactly the shape that modern coding and research agents produce: long context, repeated turns, short outputs, and cache pressure that gets worse as the session matures. The method combines asymmetric treatment of keys and values, Walsh-Hadamard rotation, QJL removal, block-scale variants, and a hardware-native FP4 path. In the deployment-oriented path, KV tensors are stored as FP4 micro-tensors with UE8M0 group scales, queries use FP8, and dequantization is folded into AMD CDNA4 scaled-MFMA operations.
The headline numbers are strong. On the paper’s long-context, multi-turn workload, UltraQuant improves P50 time-to-first-token by 3.47× in cache-pressured late rounds and 2.3× across all rounds versus an FP8 KV baseline. It raises output throughput by 1.63× over FP8 KV in the same reported table. On throughput microbenchmarks, it reaches 1.38× BF16 output throughput, essentially matching hardware FP8 KV at 1.37× while staying ahead of optimized TurboQuant variants.
The most honest number is the less flattering one: warm rounds r2-r3 report 0.86× P50 TTFT, meaning FP8 is faster before cache pressure dominates. That is not a blemish; it is the deployment policy hiding in the result. Aggressive KV compression is not a universal win. It becomes compelling when memory pressure is the limiting factor, which is precisely where long-running agents tend to end up.
There is a quality bill too. The paper reports Qwen3.5-A3B on LCB-128K dropping from 76.54 to 74.07 in one UltraQuant condition, and AIME25 dropping from 90.00 to 76.67. GPQA-Diamond and MATH500 look flatter. That unevenness is the point: the right question is not whether 4-bit KV cache is “safe.” The right question is which workload slices tolerate it, under what latency pressure, and with which fallback path.
Agent FinOps has moved below the prompt
The morning version of this cost story was TokenPilot: preserve prompt-cache continuity so providers can reuse stable prefixes instead of reprocessing rewritten context every turn. UltraQuant is the other half of the same systems problem. Even if your prompt layout is stable, the serving system still has to hold the resulting KV cache somewhere. Bigger context windows made agents more capable; they also moved the cost center into memory, cache eviction, and kernel implementation.
For practitioners, the measurement checklist should change. “Average latency” is too blunt. Track per-round TTFT, context length, cache residency, eviction rate, concurrency, output throughput, and quality deltas under realistic multi-turn traces. If your product is a coding agent, benchmark round six after it has read files, run tests, failed once, and recovered — not a pristine single prompt. If your product is an enterprise research agent, test the tenth follow-up after a long retrieval chain. The bad day will not look like your leaderboard harness.
The deployment policy should also become workload-aware. Use lower-precision KV cache where late-round latency and concurrency dominate and your evals show tolerable quality loss. Keep higher-precision cache for tasks where reasoning quality is fragile, especially math-heavy, security-sensitive, or patch-generation steps. A cheap late turn that silently degrades the answer is not cheaper; it just moves the cost into review, retries, and user distrust.
This is also why “we switched to a cheaper model” is becoming an incomplete FinOps sentence. Model choice matters, but the agent loop decides whether the infrastructure pays the same computational bill repeatedly. Prompt layout, cache billing, KV precision, GPU ISA, scheduler policy, and kernel support now sit on the same cost path as the model name. The hidden line item is not hidden to the GPU.
The broader signal is that agent infrastructure is maturing into boring systems work. Good. That is what happens when a product category stops being a demo. Long-running agents do not just need cheaper tokens; they need serving systems that keep sprawling execution state resident without turning every late turn into a latency tax. UltraQuant is not a universal answer, but it is aimed at the right layer of the stack.
Sources: arXiv, TurboQuant, vLLM, DeepSeek-V2 MLA