DeepSeek Passed GPT-5.6 in Traffic. That Doesn’t Make It Smarter.

LLM leaderboards keep inviting the wrong question. DeepSeek V4 Flash 0731 moved ahead of GPT-5.6 Luna on OpenRouter this week, but the useful conclusion is not that DeepSeek suddenly became smarter. It is that production traffic has a voting system of its own, and price, latency, availability, and routing defaults get more ballots than benchmark bragging rights.

DeepSeek's model processed 8.91 trillion tokens for the week through September 21, versus 8.48 trillion for Luna. That 430 billion-token gap pushed DeepSeek from fifth to fourth and knocked OpenAI's model down one place. It is a visible change in the top five, but only a 5.1% lead. One large application, a pricing promotion, or a routing-policy change could plausibly reverse it next week.

That caveat is not an attempt to explain away the result. It is the result. OpenRouter counts prompt and completion tokens served through its own network; private and zero-data-retention traffic is excluded, free variants are ranked separately, and providers do not all tokenize the same text identically. The chart measures consumption on a particular marketplace. It does not measure completed tasks, satisfied users, business value, or intelligence.

The preference board stayed still while deployment moved

Arena AI's overall top 20 did not change in the same period. Anthropic models still occupy six of the top ten positions, including the first three, while DeepSeek V4 Flash does not appear in that top 20. That makes Arena a useful control, even with its own selection effects: human preference remained steady while routed workload shifted.

The divergence is more informative than either ranking alone. Arena asks which answer people prefer in head-to-head comparisons. OpenRouter shows where tokens were actually spent. Production systems optimize a larger equation: acceptable quality multiplied by availability, divided by latency and cost, with reliability and integration friction lurking in the fine print. The “best” model can lose traffic if the second-best model is fast, cheap, easy to route, and good enough at the dominant task.

There is another reason token volume should not be treated as task volume: verbosity. A model that emits 25% more tokens to solve the same problem looks more popular on a token chart even if it completes fewer jobs. Agentic loops amplify the distortion because failed tool calls, retries, and self-correction all count as usage. Cost per million tokens is therefore a procurement metric, not an engineering outcome. The denominator that matters is cost per successful task.

Free Nemotron traffic is a test campaign, not a purchase order

The week's second move makes the economics even clearer. Nemotron 3 Ultra's free variant rose from eighth to seventh with 4.78 trillion tokens, passing Tencent's Hy3 at 4.51 trillion. NVIDIA describes Nemotron as a 550-billion-parameter mixture-of-experts model with 55 billion active parameters, hybrid Mamba-attention, controllable reasoning, native speculative decoding, and a one-million-token context window. Those specifications are unusually well aligned with long-running agents, synthetic-data generation, repository analysis, and batch inference.

NVIDIA also claims throughput advantages of 5.9× over GLM-5.1-754B-A40B, 4.8× over Kimi-K2.6-1T-A32B, and 1.6× over Qwen-3.5-397B-17B in a workload with 8K input and 64K output. It published checkpoints and 173 billion tokens of fresh code data. Those details give infrastructure teams something concrete to evaluate rather than another mysterious API endpoint.

But “free” is doing real work in that number. A zero-priced endpoint lowers the cost of curiosity and encourages workloads that would never survive a budget review at normal rates. Rank seven proves there is demand to experiment; it does not prove durable willingness to pay. Teams considering Nemotron should run the same load against a paid endpoint or their own serving stack, include GPU utilization and operational labor, and test behavior under concurrency. If the economics only work while somebody else subsidizes inference, the leaderboard position is marketing spend in chart form.

Gemini's tiny lead contains a useful benchmark warning

Gemini 3.8 Flash also moved up one place, processing 2.24 trillion tokens against 2.22 trillion for Meta's Muse Spark 1.3 Contributor. A 20 billion-token margin is statistical weather, not climate. Still, Google's model card offers a plausible explanation for the interest: a one-million-token context window, 64K output, configurable reasoning effort, and discounted pricing of $0.75 per million input tokens and $3.75 per million output tokens.

The benchmark details are more valuable than the rank. Google reports 73.7% on DeepSWE v1.1, up from 65.3% for Gemini 3.7 Flash, and 89.4% on Terminal-bench 2.1, up from 85.8%. Yet on Terminal-bench 4.0 it scores 19.1%, far behind Claude Opus 5 at 51.8%. “Agentic coding” is not one capability. A model can look excellent on one terminal harness and weak on a newer environment that changes task distribution, tool use, or evaluation rigor.

For engineering leaders, that spread should kill the habit of copying a public benchmark into an architecture decision. Build an evaluation set from the failures your team actually pays for: incorrect patches that pass shallow tests, destructive shell commands, dependency hallucinations, ignored repository conventions, and agents that loop until the token bill catches fire. Record task success, wall-clock latency, generated tokens, retries, and human repair time. Then compare models under the same tool permissions and stopping rules.

Use the charts as sensors, not selectors

The OpenRouter ranking has important blind spots. It misses self-hosted deployments, direct-provider API traffic, and subscriptions such as Claude Code and Codex. Its users are not a random sample of the software industry; the marketplace attracts indie builders, researchers, experimentation-heavy applications, and workloads that benefit from model switching. Community discussion around OpenRouter's broader token studies has also raised privacy concerns about prompt classification and noted that routing a small model through an aggregator can be irrational when it fits on consumer hardware. Others correctly counter that hosted throughput still wins for bursty batch jobs and multi-agent systems.

The practical response is not to dismiss public rankings. It is to assign each one the right job. Treat OpenRouter as a demand sensor, Arena as a preference sensor, and model cards as hypothesis generators. None should be your model selector.

If DeepSeek's advantage over Luna widens for three to seven snapshots, there is a real deployment trend worth investigating. If Nemotron holds its position after free capacity or pricing changes, its architecture may be translating into sustainable economics. If Gemini stays above Muse rather than swapping places tomorrow, the Flash proposition may be winning repeat workloads. Until then, these moves are candidates for an internal bake-off, not reasons to rewrite the production router.

DeepSeek did not “beat OpenAI” this week. It won more metered traffic on one marketplace by a modest margin while the human-preference leaderboard stood still. That is less dramatic than a benchmark upset and more useful to builders: model adoption is increasingly decided in the messy middle where quality is sufficient, latency is tolerable, and the invoice can survive contact with scale.

Sources: OpenRouter rankings, OpenRouter methodology, NVIDIA Nemotron 3 Ultra, Google Gemini 3.8 Flash model card, Hacker News discussion