Quantized Reasoning Models May Be Cheaper Per Token and Slower Per Answer

The comfortable story about quantization is that smaller weights make inference cheaper. For ordinary generation, that story is often close enough. For reasoning models, it has a missing column: the model may become cheaper per token and more expensive per answer.

A new paper on low-bit reasoning models identifies a serving cost that most dashboards still hide. Quantization can preserve final-answer accuracy while making the model generate longer chains of thought. The authors call the metric CoT Token Inflation Ratio, or CTIR. The acronym is not glamorous, which is fine. The metric is the useful part. If a quantized reasoning model emits 40 percent more hidden reasoning tokens to reach the same answer, its per-token speedup may be eaten by extra work before the user sees anything.

The study evaluates Qwen3-4B-Thinking-2507, Phi-4-Reasoning-Plus, and Qwen3-30B-A3B-Thinking across AIME25, GSM8K, GPQA-Diamond, BBH, LiveCodeBench, MBPP, HellaSwag, PIQA, and MMLU-Pro. It tests RTN, GPTQ, AWQ, and HQQ at INT4 and INT3, with BF16 as the reference. The familiar headline is there: INT4 can preserve accuracy on larger models. Qwen3-30B BF16 averages 74.8 on reasoning tasks, while INT4 RTN reports 75.6 and INT4 GPTQ reports 74.6 in the paper's table.

That is exactly the sort of result that gets quantized models approved in procurement meetings. Accuracy looks flat. Memory use goes down. Tokens per second look better. Everyone nods, and the dashboard has no idea that the model is thinking longer.

The hidden tokens still hit the GPU

The CTIR numbers are the real story. Qwen3-4B INT4 RTN has an average CTIR of +42.5 percent. Qwen3-4B INT3 AWQ reaches +276.1 percent. Qwen3-4B INT3 GPTQ increases MBPP reasoning tokens from 4,692 in BF16 to 9,704, a 106.8 percent increase. The larger model is not immune: Qwen3-30B INT4 RTN has an average CTIR of +43.0 percent, while Qwen3-30B INT3 AWQ still shows +32.8 percent.

These tokens may be hidden from the user, but they are not hidden from the server. They occupy decode time, consume KV cache, reduce batching efficiency, and push out time-to-first-visible-token. The serving experiment in the paper reports that a 1.1x CTIR increase raises p90 time-to-first-visible-token from 1050 seconds to 1400 seconds, a 33 percent tail-latency increase, while throughput drops from 0.055 to 0.047 requests per second. Even if those absolute timings are specific to the experimental setup, the direction is the warning: per-token efficiency is not the same as per-task efficiency.

This matters most for the exact workloads where teams are now reaching for reasoning models: coding agents, math solvers, multi-tool workflows, planning-heavy assistants, and long-context debugging. In those settings, the bill is not just visible output tokens. The model may spend a large hidden budget checking itself, restating constraints, exploring false starts, and recovering from uncertainty. If quantization perturbs the trajectory into extra self-checking loops, a model can be both accurate and operationally worse.

The BFCLv4-Multiturn result is a good example. GPTQ-INT4 increases Qwen3-4B chain-of-thought tokens by 13.3 percent while tool calls are nearly unchanged, from 7.56 to 7.85. The model is not obviously doing more external work. It is spending more internal reasoning budget around roughly the same tool behavior. If you only measure tool-call count or final accuracy, you miss the cost shift.

Prompting is not a cost-control strategy

The mitigation section is useful because it punctures the lazy fix. Prompting the model to be concise can reduce reasoning tokens, but it can also remove the reasoning that kept the answer correct. The paper reports that a “no restate” prompt cuts INT4 MBPP chain-of-thought tokens by 30.0 percent but drops accuracy from 90.0 percent to 81.0 percent. For INT3 prompts, the paper sees 5 to 11 point accuracy drops. That is not optimization; that is squeezing the model until it stops doing the useful part.

Calibration data matters more. Switching from Wikitext to Math-CoT raises accuracy from 28 percent to 65 percent while cutting chain-of-thought length from 30k to 12k tokens for Qwen3-4B INT3 GPTQ. That is a strong reminder that reasoning models have activation patterns that generic text calibration may not represent. If the quantizer never sees reasoning-like distributions, it should not be surprising when the quantized model wobbles during reasoning.

Quantization-aware training is the strongest mitigation in the paper: 68 percent accuracy at 9.5k average chain-of-thought tokens, close to BF16's 73 percent and 8.5k tokens. The lesson is straightforward. If reasoning behavior is part of the product, quantization has to preserve trajectories, not just approximate weights after the fact. You cannot validate a reasoning model solely by asking whether the last token lands on the right answer.

For practitioners, the action list is concrete. Add hidden reasoning-token accounting to model evals. Compare BF16 and quantized variants on end-to-end latency, p90 and p99 time-to-first-visible-token, success rate, total generated tokens, KV-cache pressure, and cost per correct answer. Segment by workload. A quantized model that works well for short answers may be bad for code repair. A model that is fine on GSM8K may inflate badly on LiveCodeBench. A model that looks cheap in a provider rate card may be expensive inside an agent loop.

Procurement should change too. Do not approve an INT4 or INT3 reasoning model because benchmark accuracy is flat and memory use is lower. Ask for request-level economics. How many total tokens per solved task? How does tail latency move? Does calibration data match your domain? Does the model preserve tool-use behavior and not just final-answer accuracy? Can you set budget controls based on reasoning effort without quietly deleting correctness?

The take: INT4 is not a free lunch for reasoning. Quantization can move cost from memory bandwidth into hidden reasoning length, and most production dashboards are not watching that column. The cheapest model is the one with the best cost per correct answer under your latency SLO, not the one with the smallest weights.

Sources: arXiv, GPTQ paper, AWQ paper, vLLM project