The LLM Leaderboard Is Becoming a Distribution Leaderboard
Model leaderboards increasingly tell us less about which model is best and more about which model is easiest to try. This week’s OpenRouter chart makes that distortion unusually visible: Tencent’s production Hy4 held fourth place while its preview version fell four spots, and an anonymous free model called Crouton Alpha jumped from fourteenth to tenth. Trillions of tokens moved, but quality did not necessarily move with them.
That distinction matters because OpenRouter’s ranking is often read like a performance table. It is not. The chart counts prompt and completion tokens routed through one gateway, grouped into daily UTC buckets and separated by model variant. It does not measure unique users, successful tasks, dollars spent, latency, correctness, or how much human cleanup followed each response. Private requests are excluded. A model can rise because it is excellent, because it is free, because it is verbose, because it accepts enormous prompts, or because a provider made it the default in a popular application.
The top of the chart did not change. DeepSeek V4.1 Flash 1017 remained first with 56 trillion weekly tokens, up 17%, followed by DeepSeek V4.1 Flash at 17.6 trillion. GLM 5.3 Flash held third at 13.2 trillion despite a 12% weekly decline. Hy4 stayed fourth with 12.8 trillion tokens, ahead of OpenAI’s GPT-6 Luna at 7.21 trillion. Stability at the top makes the movement below it more useful: this is a week about migration and distribution, not a new champion.
Hy4’s “decline” is really a production migration
Hy4 Preview dropped from ninth to thirteenth as its weekly volume fell 61% to 3.51 trillion tokens. Viewed alone, that looks like rejection. Viewed beside the production Hy4 endpoint, which grew from 10.1 trillion to 12.8 trillion tokens in the trailing window, it looks like the normal handoff from preview to release. Together the two endpoints carried 16.31 trillion tokens—enough to rank behind only the two DeepSeek entries if OpenRouter grouped model families rather than SKUs.
This is the first practical lesson for anyone building internal adoption dashboards: group versions before declaring winners and losers. A release that shifts traffic from model-preview to model can manufacture a falling model and a rising model from the same customer base. Routing telemetry should preserve the exact model identifier for reproducibility, but reporting should roll it up by family when the question is adoption.
Tencent is also pitching Hy4 as more than a chat endpoint. The company says it supports native vision, computer use across command-line and graphical interfaces, and specialized work in Blender and CAD, with integrations spanning Unreal Engine, Unity, and Godot. Its own WorkBuddy evaluation reports an average score of 83.18, a 96.15% task pass rate, and first place in nine of twelve mainland-China scenarios. A separate Tencent blind evaluation involved 213 financial professionals and 579 tasks; Hy4 scored 3.04, versus 2.97 for DeepSeek V4.1 Flash 1017 and 2.68 each for Kimi K3.1 Pro and GLM 5.3.
Those are specific and potentially useful numbers, but they remain vendor-run results on product-specific harnesses. Tencent further reports hallucinations falling from 12.95% in Hy4 Preview to 7.2%, unsupported assumptions falling from 23.1% to 5.6%, and improvements of 19.1% on GameDev and 9.8% on GameLogic. The breadth is impressive; it is also a warning label. A model tuned into a complete agent stack can excel inside its maker’s tools while behaving differently with your schemas, prompts, retries, and permission boundaries.
Hy4’s economics are competitive enough to encourage experimentation. Tencent’s direct prices are roughly $0.90 per million input tokens, $2.70 per million output tokens, and $0.045 per million cached input tokens. OpenRouter currently lists a Tencent Cloud route at $0.417 per million input and $1.251 per million output. That gap should not be treated as a permanent discount. Aggregator pricing can reflect promotions or alternate routes, and provider selection can affect rate limits, data handling, latency, and reproducibility. Hy4 supports tool use, structured responses, reasoning controls, and streaming on OpenRouter, but it is not currently offered through EU in-region routing.
Crouton Alpha is what subsidized discovery looks like
Crouton Alpha’s four-place jump is the sharper illustration of leaderboard incentives. The model processed 4.27 trillion weekly tokens, is free, and remains anonymous. OpenRouter metadata indexed by search describes a 1,048,576-token context window and positions it for coding, agentic, and multimodal work. That combination is practically designed to maximize token volume: developers can pour repositories, transcripts, screenshots, and long agent traces into it without watching a meter.
The demand is real, but its meaning is ambiguous. Free access converts curiosity into enormous prompt volume, especially when context is measured in a million tokens. A long-context model can climb faster than a concise specialist even when fewer people use it and fewer jobs finish successfully. This is the second original read on the chart: token rankings systematically reward architectural appetite. They give more credit to consuming the work than completing it.
An undisclosed provider introduces a separate governance problem. Teams should not send proprietary code, customer records, regulated data, or production credentials to an endpoint until ownership, retention, training use, subprocessors, residency, and incident-response terms are explicit. Crouton Alpha also lacks EU in-region routing. “Anonymous and free” is a reasonable sandbox proposition; it is not a production procurement profile.
Kimi K3.1 Pro offers a useful comparison. It rose one place to ninth on 4.37 trillion tokens and 91% week-over-week growth—almost the same volume as Crouton Alpha, but without the same four-rank leap. Elsewhere, GPT-5.6 Luna fell from fifteenth to nineteenth as volume dropped 36% to 2.75 trillion. These moves may reflect application defaults, pricing changes, migrations, or promotions as readily as model preference. OpenRouter traffic is a live map of routing behavior, not a controlled experiment.
Run a bake-off that measures completed work
Engineers evaluating Hy4 or Crouton Alpha should put both behind an abstraction layer and replay a fixed corpus of representative jobs. Measure task completion, tool-call validity, schema compliance, retries, wall-clock latency, input and output tokens, and cost per successful task. Include failure cost: a cheap response that requires ten minutes of review is not cheap. Log provider and model-version identifiers, pin routing when reproducibility matters, and test fallback behavior deliberately rather than discovering it during an outage.
For agentic systems, include adversarial cases: malformed tool output, unavailable tools, permission denials, stale UI state, and prompts that mix trusted instructions with untrusted page content. For long-context work, compare retrieval against brute-force context stuffing. A million-token window may simplify a prototype while increasing latency, distraction, and exposure of unnecessary data. Context capacity is a ceiling, not a target.
Finally, set a spend ceiling and a migration trigger before adopting any free endpoint. Record the price and service terms assumed by your evaluation, then decide what increase would force re-testing. Free tiers are acquisition channels, not architecture. If a system only works economically while someone else subsidizes trillions of tokens, the design has an expiration date.
Arena’s human-preference leaderboard was unavailable for this run because its collection path returned no rows and the fallback hit Cloudflare verification. That absence limits the claim we can make—and improves the honesty of the one we can. This week shows production migration and subsidized discovery at scale. It does not show that Hy4 or Crouton Alpha is “better.” The model leaderboard is becoming a distribution leaderboard, and engineering teams should read it as market telemetry before they read it as technical evidence.
Sources: OpenRouter weekly rankings, Tencent’s Hy4 announcement, Hy4 API and pricing, Crouton Alpha API and pricing, Hacker News discussion, LocalLLaMA discussion