STARE Finds the Token-Level Bug Behind GRPO Entropy Collapse

Entropy collapse is one of those training failures that sounds abstract until it burns a run. The reward curve looks promising, the model starts getting better, and then exploration quietly narrows. The policy becomes more confident before it becomes sufficiently competent. STARE is useful because it treats that failure not as mystical RL instability, but as a token-level credit-assignment bug hiding inside GRPO-style training.

GRPO assigns a trajectory-level advantage. That is elegant, cheap, and part of why the post-DeepSeek reasoning-model boom moved so quickly. But language models do not update in trajectory-sized chunks. They emit tokens, and different tokens have different effects on entropy. Some are boring, high-probability continuations that make the policy more deterministic when reinforced. Others are rarer exploratory moves that preserve or increase diversity. If the whole trajectory gets a positive advantage, the update reinforces both. The common tokens win by frequency. Exploration shrinks while the dashboard still says “learning.”

STARE — Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability — proposes a targeted fix. It operates inside GRPO’s clipped surrogate objective, identifies entropy-critical token subsets using batch-internal surprisal quantiles, and reweights advantages for those tokens. In the O1 variant, it amplifies positive-advantage high-surprisal tokens. In the C2 variant, it also attenuates negative-advantage high-surprisal tokens. A target-entropy gate turns the intervention on when batch mean entropy falls below H_tgt and backs off when entropy recovers.

This is control theory, not vibes

The default hyperparameters are refreshingly specific: learning rate 1e-6, batch size 64, eight rollouts per sample, top-p 1.0, temperature 1.0, STARE weights W=1.1 and M=0.9, target entropy H_tgt=0.3, and top surprisal quantile P=10%. The paper evaluates Short CoT, Long CoT, and Multi-Turn Tool Use across models from 1.5B to 32B, including Qwen2.5-Math-7B, Qwen2.5-14B, Qwen2.5-32B, DeepSeek-R1-Distill-Qwen-1.5B, Qwen3-8B, and a Qwen2.5-7B tool-use setup cold-started from Retool 2K.

The benchmark set is math-heavy: AIME24, AIME25, AMC23, MATH-500, Minerva Math, and OlympiadBench. On Qwen2.5-Math-7B, STARE-O1 reaches a 54.4 average versus 49.1 for GRPO-ds and 47.2 for DAPO. On Qwen2.5-14B, STARE-C2 reaches 52.3 versus 46.1 for GRPO-ds. On Qwen2.5-32B, it reaches 61.4 versus 56.1. In the Long-CoT DeepSeek-R1-Distill-Qwen-1.5B setting, STARE-C2 reaches 66.3 versus 62.5 for GRPO-ds and 55.1 for DAPO. In the tool-use agent scenario, STARE-C2 reaches 60.4 average versus 53.9 for GRPO-ds, including 52.8 on AIME24 and 38.1 on AIME25.

The stability claim may be more important than any one score. The authors say STARE sustains more than 5,000 RL steps on 7B models and more than 1,500 steps on 14B and 32B models while keeping entropy within the target band. That is the kind of detail model builders should demand from reasoning-model papers. A leaderboard number without training stability is a screenshot, not an engineering result.

For practitioners, the action item is instrumentation. If you are doing RL post-training, reward curves and final benchmark accuracy are not enough. Track token-level entropy, rollout diversity, within-group variance, response length, high-surprisal token behavior, and whether improvements are coming from genuine reasoning or from memorized short paths. Watch for reflection-token inflation too. Reasoning models can fail by collapsing too early, but they can also fail by rambling longer without solving harder problems. The point is to know which failure you are buying.

Teams evaluating vendor models should care as well. Users experience “better reasoning” as a product feature, but the feature is shaped by invisible training geometry. A model that looks sharper after RL may also be less exploratory, more brittle under prompt shifts, or more prone to overconfident shortcuts. When vendors talk about RL improvements, ask for more than benchmark deltas: variance across seeds, regressions on non-target tasks, length behavior, calibration, and how the method performs outside math.

The cost angle is subtle. Stable RL methods can reduce wasted runs by avoiding collapse. They can also tempt teams to run longer because the training loop looks healthier. The right metric is improvement per training dollar, not just final AIME score. For agentic systems, the same discipline applies: measure whether post-training improves tool use, retrieval decisions, safety refusals, and code editing under realistic harnesses, not only math contest performance under a friendly prompt.

The limitation is that STARE’s evidence is still concentrated around math and reasoning benchmarks. The inclusion of a tool-use setup is welcome, but production agents have uglier entropy problems: choosing whether to call a tool, which file to inspect, when to ask for clarification, and when to stop. Those choices are not always captured by token surprisal in a clean way. STARE’s mechanism is promising; its deployment story still needs broader agent benchmarks.

Still, this is exactly the sort of paper worth reading because it explains a mechanism rather than selling a model name. The next wave of reasoning models will be won by details like this: not just better rewards, but better control over what the policy becomes while chasing them. Boring training geometry is where the product improvements come from. The product page just gets the nicer font.

Sources: arXiv, STARE code, DeepSeekMath / GRPO, DeepSeek-R1, Qwen