On-Policy Distillation Looks Dense on Paper, but the Parameter Updates Are Sparse Where It Counts

On-policy distillation sounds like it should rewrite models broadly. The student generates its own trajectories, the teacher provides dense token-level feedback, and the optimizer gets a lot more signal than sparse reward RL. The surprise in this paper is that the final parameter updates still look selective.

“Dense Supervision, Sparse Updates” analyzes what on-policy distillation changes inside language and vision-language models. The authors compute checkpoint deltas between source models and OPD-trained models, then inspect scale, coordinate sparsity, layer/module energy, spectral concentration, and alignment with the source weights' principal directions. The result is a useful correction to a lazy assumption: dense teacher feedback does not necessarily mean dense model editing.

OPD sits in an important middle lane in modern post-training. Supervised fine-tuning and offline distillation train on fixed demonstrations, which creates distribution-shift problems when the model later generates its own imperfect prefixes. RLVR trains on the current policy's own samples but often learns from sparse outcome rewards, which creates credit-assignment pain. OPD combines on-policy student behavior with dense teacher supervision. The student acts in its own distribution; the teacher provides token-level feedback on those student-generated trajectories.

That hybrid design raises a weight-space question. Does OPD look more like supervised distillation because the feedback is dense? Or more like RL-style post-training because the data is on-policy? This paper says the on-policy part matters more than many teams may expect.

The model moves less than the objective suggests

Across the analyzed OPD-style pairs, the paper finds small relative checkpoint deltas, high visible coordinate sparsity, distributed layer updates, and usually FFN-heavy energy. The abstract's compressed version is clean: OPD updates are small, coordinate-sparse, distributed across layers, numerically full-rank but spectrally concentrated, and biased away from the source weights' principal singular subspaces. That last point matters. The model is not simply turning the big existing directions louder. It is writing into less dominant coordinates, often where source weights are close to zero.

The numbers make the contrast sharper. OPD-style relative Frobenius norms range from 0.036% for one Qwen3 OPSD case to 0.142% for a Qwen2.5-VL OPD case, with 66.72% to 89.50% visible coordinate sparsity at a 1e-5 threshold. The offline distillation contrast is very different: 11.94% relative delta norm and only 3.06% sparsity. That is not a small implementation detail. It says OPD can produce capability changes without behaving like ordinary dense parameter rewriting.

Module energy also has a recognizable shape. In several OPD runs, FFN modules account for 65.06% to 85.88% of delta energy. Qwen3 OPSD is a useful exception, with attention contributing 37.36%, and Qwen3-1.7B OPD also showing meaningful attention energy at 27.03%. So the lesson is not “only FFNs matter.” The lesson is that OPD edits are structured, and the structure is measurable.

The JustRL boundary case is especially useful. A JustRL checkpoint shows larger relative delta norm, 0.497%, and lower visible sparsity, 34.68%. But when that JustRL checkpoint is used as the OPD teacher, the resulting OPD student returns to the usual pattern: relative norm 0.108%, sparsity 70.43%, plus the same spectrally concentrated, off-principal behavior. In other words, the teacher's own sparsity is not required for the student update to look sparse.

Why practitioners should care about checkpoint geometry

Most product teams will never run this analysis directly on a frontier-scale model. That does not make it academic trivia. Post-training mechanics are becoming product mechanics. When a vendor says a reasoning model improved through OPD, RLVR, GRPO, or teacher distillation, the final benchmark number is only one surface. The hidden question is what changed inside the model and what kind of generalization or interference risk that change implies.

Sparse update structure opens practical doors. It suggests that cheaper OPD variants, mask discovery, targeted adaptation, and checkpoint-delta diagnostics may recover much of the performance of full training. The paper reports that retraining only the discovered update subnetwork can recover nearly the same performance as full OPD, while density-matched random masks underperform. That is the kind of result that matters if you are trying to adapt smaller reasoning models without paying for broad full-parameter churn every time.

But the optimizer result blocks an overly cute conclusion. Sparse final deltas do not mean a sparsity-inducing optimizer is enough. The paper's optimizer ablation finds that SGD underperforms AdamW in OPD, even though SGD can induce sparsity. Dense teacher supervision preserves heterogeneous coordinate-wise gradient scales, and adaptive scaling still helps. Translation: do not cargo-cult RLVR optimizer folklore into OPD just because the final mask looks sparse. Different objective, different gradient structure, different bill.

The action item is diagnostic discipline. If your team runs OPD, teacher-driven post-training, or RLVR, treat checkpoint deltas as an evaluation artifact. Inspect relative norm, coordinate support, layer distribution, FFN versus attention energy, spectral concentration, and overlap with earlier training masks. Compare those signals against benchmark gains and regression tests. A model that improves one math benchmark through a tiny FFN-heavy subnetwork may carry a different robustness profile than one that moves broadly across attention and FFN modules.

This also connects to model governance. As more capability comes from post-training rather than pretraining alone, “what data was used?” is not enough. Teams need to know how the model was changed. Did the update write broadly? Did it target a discoverable subnetwork? Did it move away from source principal directions? Did it overlap with prior safety or capability masks? These questions sound esoteric until a narrowly improved model regresses in an adjacent domain and nobody has the diagnostics to explain why.

The paper's strongest contribution is not a new training method. It is a better instinct: benchmark scores should come with weight-space receipts. OPD looks dense at the supervision layer, but its parameter signature still carries the fingerprints of on-policy learning. That is exactly the kind of invisible mechanics frontier-model buyers and builders need to start asking about.

Sources: arXiv, project repository, DeepSeekMath/GRPO background, DeepSeek-R1 RL reasoning background