The LLM Leaderboard Is Becoming a Cost Curve

The most useful LLM leaderboard this week is not really a leaderboard. It is a cost curve with model names attached.

OpenRouter's trailing seven-day usage chart puts OpenAI's GPT-5.6 Luna first with 17.3 trillion tokens, followed by Tencent's Hy4 preview at 12 trillion. Z.ai's GLM 5.3 Flash has edged into third with 11.3 trillion, just ahead of DeepSeek V4 Flash 0731 at 11.2 trillion. The 100-billion-token gap sounds enormous until you divide it by the scale of the workloads involved: GLM's lead is less than 1%, small enough for one large batch pipeline or routing-policy change to reverse it.

That narrow swap is not the story. The composition of the top ten is. Seven of the nine slots below Luna belong to Chinese providers: Tencent, Z.ai, DeepSeek, and Xiaomi. DeepSeek alone occupies fourth, fifth, and eighth place; Xiaomi's MiMo-V2.5 sits sixth with 7.83 trillion tokens; Tencent's Hy3 rose to seventh with 4.47 trillion. Nvidia's free Nemotron 3 Ultra is ninth, and Z.ai's full GLM 5.3 moved into tenth.

This is what the market looks like when developers can switch models with an API parameter and have to pay the invoice afterward. Prestige matters. Price, throughput, availability, and the probability that a tool call actually completes matter more.

Luna's lead is a deployment decision, not a quality crown

OpenRouter ranks models by tokens processed, not requests, unique users, revenue, or independently measured quality. It aggregates prompt and completion tokens into trailing seven-day UTC buckets, excludes private usage, and counts variants separately. Direct traffic to OpenAI, Anthropic, Google, DeepSeek, and the major cloud platforms is invisible. Different tokenizers and different output habits further muddy comparisons: a verbose model can climb the chart while doing no more useful work.

Even with those caveats, 17.3 trillion third-party tokens are a strong adoption signal. GPT-5.6 Luna is explicitly built for high-volume, cost-sensitive workloads. OpenAI lists it at $0.20 per million input tokens, $0.02 per million cached input tokens, and $1.20 per million output tokens. It supports a 1.05-million-token context window, outputs up to 128,000 tokens, structured responses, function calling, prompt caching, web search, code execution, MCP, and computer use.

That feature set matters because the cheap-model tier is no longer limited to classification and autocomplete. A low-cost model can now sit inside an agent loop, call tools, emit schema-valid objects, and carry a large working set. Luna's input price matches GPT-5.4 nano, while its output price is five cents lower per million tokens. OpenAI also reports that responses containing at least one factual error were roughly 62% less common than with GPT-5.5 Instant in its internal financial, medical, and legal evaluations. That is vendor-reported and task-specific, not a universal reliability score, but it helps explain why teams might treat Luna as infrastructure rather than a toy tier.

There is an expensive footnote. Requests exceeding 272,000 input tokens are charged at twice the input rate and 1.5 times the output rate across the entire request. An agent harness that keeps appending its whole transcript can cross that boundary quietly. The advertised bargain then becomes an architecture tax, especially when repeated tool traces and retrieved documents are mostly dead context.

The Chinese-model cluster is the stronger signal

The leaderboard's second through eighth positions show a market where several providers have cleared the "good enough to route production traffic" threshold at once. That is more consequential than any one model taking third place. When the quality floor rises across vendors, the differentiator shifts from benchmark peaks to cost per successful task, regional availability, latency, cache behavior, and operational predictability.

DeepSeek V4.1 Flash illustrates both the opportunity and the risk. It reached fifth place with 10.3 trillion weekly tokens roughly a week after its September 10 release. But DeepSeek also announced that V4 Pro requests would temporarily route to V4.1 Flash from September 14 until V4.1 Pro launches. Some of that rapid adoption is likely a provider migration rather than millions of developers making an explicit model choice.

That distinction is not pedantic. If a provider can move traffic behind a familiar endpoint, a model name in application configuration is not necessarily the model that generated a production response. Teams in regulated, safety-sensitive, or reproducibility-heavy environments should pin snapshots where possible, log the resolved model identifier, and canary every provider-side change. The leaderboard can detect traffic movement; it cannot tell you whether that movement was voluntary.

A second implication is that marketplace rankings increasingly measure routing systems as much as model demand. OpenRouter and its users can optimize for price, provider health, latency, and availability. At this scale, a default change in an SDK, a free promotion, or a large customer's batch job can move trillions of tokens. Usage rank is evidence of deployment economics, not a democratic vote on intelligence.

Stop selecting one model; build a small portfolio

Arena's visible overall top 20 did not change in the same snapshot. Anthropic holds seven of those positions, including the first three, while the OpenRouter usage chart tells a very different story. That divergence is useful. Arena captures comparative human preference under its testing setup; OpenRouter captures traffic through one marketplace. Collapsing both into a single idea called "best model" discards the information engineers actually need.

The practical response is not to replace one benchmark winner with this week's usage winner. Maintain a small routing portfolio. Use a cheap default for extraction, classification, summarization, retrieval cleanup, and routine agent turns. Escalate ambiguous, high-risk, or low-confidence cases to a stronger model. Keep a pinned fallback from another provider so an outage or silent regression does not stop the product.

Then evaluate the portfolio on your own traces. A useful starting point is 100 to 500 representative tasks run against Luna, GLM 5.3 Flash, and DeepSeek V4.1 Flash. Record acceptance rate, structured-output validity, tool-call success, retry count, p50 and p95 latency, cache hit rate, input and output volume, and total cost per accepted result. Include failure handling. A model that costs 20% less per token but produces 30% more output, retries tools twice, or sends more cases to human review is not cheaper.

For agent workloads, test context growth explicitly. Replay short, medium, and pathological sessions; summarize or discard stale tool output; and alert when prompts approach pricing thresholds. For routing changes, shadow a fraction of production traffic before cutover and compare semantic outcomes, not just HTTP success. Store prompts, model snapshots, resolved provider IDs, and evaluation versions so that a regression can be reproduced rather than debated in Slack.

The final lesson from this week's chart is pleasantly unglamorous. Models win production traffic by disappearing into the stack: cheap enough to call repeatedly, reliable enough not to babysit, fast enough to stay off the critical path, and replaceable enough that a team is not trapped. Luna currently leads that race, while Tencent, Z.ai, DeepSeek, and Xiaomi are turning the next nine slots into a price-performance knife fight.

The ranking will move again. The durable engineering decision is to make sure your architecture can move with it—based on measured cost per successful task, not benchmark crowns, marketplace volume, or provider vibes.

Sources: OpenRouter LLM Rankings, OpenAI GPT-5.6 Luna API documentation, OpenAI's GPT-5.6 update, DeepSeek V4.1 Flash announcement, Arena AI leaderboard