The LLM Leaderboards Disagree—and That’s the Point

The most useful thing about this week’s model rankings is that they disagree. Arena’s preference tests put Anthropic and Google’s newest releases straight into the top eight, while OpenRouter’s usage chart is being led by a different set of models—and by economics that a blind comparison cannot see. That is not noise to be averaged away. It is evidence that “the best model” has become an underspecified question.

Claude Fable 5.1 (Max) debuted at number three on Arena, behind Claude Fable 5 and Claude Opus 4.6 (High). Gemini 3.8 Flash (High) entered at number eight. Their arrival pushed every previous model from eighth through twentieth down two places, with Gemini 3.6 Flash (High) and Qwen 3.8 Max leaving the top 20. Anthropic now owns five of Arena’s first seven positions, while Google’s new Flash release has landed ahead of Claude Opus 5 (High), Meta’s Muse Spark 1.1, and its own 3.7 Flash predecessor.

Those are meaningful preference signals, but they are not a clean quality league table. Arena’s current overview exposed relative category ranks to the research scraper, not reliable Elo scores or vote totals. Without uncertainty intervals and sample counts, a rank is a useful invitation to test, not a statistically complete verdict. A new release can also attract disproportionate attention while its evaluation mix is still settling.

Preference winners are not the models eating the workload

OpenRouter’s chart describes another market. GPT-5.6 Luna rose two places to lead with 11.6 trillion weekly tokens. GLM 5.3 Flash held second and DeepSeek V4 Flash 0731 slipped from first to third, both at 11.4 trillion. The 0.2-trillion gap sounds enormous in isolation, but it is less than two percent of the leaders’ volume. That near tie is more informative than the ordinal podium: a modest routing change can swap first and third without any model becoming smarter overnight.

Tencent’s Hy4 preview climbed from sixth to fourth with 7.99 trillion weekly tokens. MiniMax M3’s free variant moved from tenth to eighth, Nvidia’s free Nemotron 3 Ultra held ninth, and DeepSeek V4 Pro 0813 entered at twentieth. Anthropic’s Claude Opus 5 and Sonnet 5 sit at eighteenth and nineteenth. Arena says users prefer a great deal of Anthropic output; OpenRouter says public marketplace traffic is flowing heavily toward OpenAI, Z.ai, DeepSeek, Tencent, and subsidized variants.

Neither chart invalidates the other because the measurement units are different. Arena approximates which answer a voter prefers in a comparison. OpenRouter counts public prompt and completion tokens in daily UTC buckets, separating free and paid variants and excluding private traffic. It does not measure users, requests, accuracy, revenue, or spend. A verbose reasoning model working through a large repository can generate more tokens than thousands of short interactive chats. A free endpoint can attract experimentation that disappears when the subsidy does. Token share is workload gravity, not a product-quality score.

The price card is no longer the bill

Claude Fable 5.1 illustrates why procurement by sticker price is increasingly unreliable. It costs $10 per million input tokens and $50 per million output tokens, which looks expensive beside the Flash class. But Anthropic cut cache reads by 75 percent to $0.25 per million tokens and estimates that this reduces typical workload cost by 25 percent and highly agentic workload cost by as much as 45 percent. One launch customer described Fable as roughly twice as fast as Opus 5 while consuming half as many tokens; Cognition said it was moving Devin’s Opus 5 traffic at launch.

If those gains survive independent testing, the expensive output rate may be the wrong number to optimize. Long-running coding agents repeatedly ingest the same repository context, plans, tool results, and policy instructions. Cheap cache reads and fewer failed attempts can dominate the cost equation. There are operational footnotes, too: default Fable use requires 30-day data retention for safety monitoring, and some cyber or biology prompts may be routed to less capable Opus models. Teams need to configure Anthropic’s fallback behavior and test it explicitly; a model migration is not reproducible if policy routing silently changes the model underneath the job.

Gemini 3.8 Flash occupies the opposite end of the visible price spectrum at $0.75 per million input tokens and $3.75 per million output tokens, unchanged from 3.7 Flash’s introductory rate. Google also says the model “works harder,” taking more reasoning steps and making iterative tool calls. That candor matters. The per-token price can stay flat while the per-task bill rises because the model reads more context, calls tools more often, or loops before converging.

Google reports 54.9 percent on HLE-Verified. Its restricted 3.8 Flash Cyber variant exceeded 70 percent success on an internal vulnerability-discovery set spanning 20 languages and reached 47.2 percent pass@1 on CWE-Bench, just behind a leading frontier model at 47.8 percent. Chrome Security says it produced 2.6 times more correct patches than the best larger commercial models in its test, while Wiz reports 7.5 to 9.7 percentage points higher recall at 2.3 to 5.2 times lower cost. These are promising, vendor-selected results—not a substitute for reproducing performance against your languages, repositories, and definition of a correct fix.

Open weights still come with an infrastructure invoice

Hy4’s OpenRouter rise is the clearest example of capability meeting distribution and price. Tencent describes it as a 770-billion-parameter mixture-of-experts model with 49 billion active parameters and a context window beyond one million tokens. Pricing is $0.834 per million input tokens, $2.501 per million output tokens, and $0.042 per million cached tokens. Tencent’s internal blind evaluation—163 experts across 203 engineering tasks—gave Hy4 an average 2.99 out of 4.00, compared with 2.94 for Kimi K3 and 2.92 for GLM 5.3.

The open-weight label does not make 770 billion parameters cheap to operate. Even with only 49 billion active on each token, memory capacity, expert routing, interconnect bandwidth, batching, and tail latency remain serious systems problems. Tencent says the model helped optimize its own inference stack and improved end-to-end throughput by 31.8 percent over baseline. That is both an impressive result and a warning: the model’s attractive hosted price is partly the product of infrastructure work most teams will not reproduce by downloading weights.

Community discussion reflects this split. Early Fable users on Hacker News praised writing and instruction-following while others complained about verbosity—an especially relevant criticism when output tokens cost $50 per million. Gemini discussion praised the cadence of Flash releases but questioned benchmark optimization and pointed out that weak agent harnesses can bottleneck a strong base model. LocalLLaMA’s Hy4 debate focused less on the 770B headline than on whether practitioners can serve 49B active parameters efficiently. These are qualitative signals, not representative surveys, but the questions are better than another round of benchmark victory laps.

Benchmark the job, not the model card

Engineering teams should use these rankings to choose evaluation candidates, not production winners. Build a shadow suite of 30 to 50 representative tasks, including ordinary work and the ugly cases that trigger retries: cross-file refactors, ambiguous bug reports, tool failures, long context, and security-sensitive prompts. Pin exact model IDs and reasoning levels. Record task success, reviewer acceptance, wall-clock latency, uncached and cached tokens, tool-call count, retry rate, and human correction time.

The key metric is verified work per dollar. A cheap model that needs three attempts and a senior engineer’s cleanup can lose to an expensive model on total cost. A premium model that reads mostly cached repository context may beat its list price. For agent workloads, report percentiles rather than averages: one runaway tool loop can matter more operationally than ten cheap successes. Run security cases separately because access controls, retention, refusals, and fallback models are part of the system’s behavior.

Version management now deserves the same discipline as dependency management. With two new models landing in Arena’s top eight and model families updating rapidly, quarterly benchmarks are too slow and aliases such as “latest” are hostile to reproducibility. Pin versions, keep golden outputs, canary upgrades, watch cost and correctness regressions together, and preserve a rollback path.

The market is not converging on one leaderboard. It is splitting into preference, capability, distribution, and unit-economics markets, each producing a different winner. Arena tells you what deserves attention. OpenRouter tells you where public tokens are moving. Neither can tell you what survives your production constraints. The only ranking worth treating as authoritative is the one built from your own verified tasks—and this week’s charts provide a very good shortlist for the next run.

Sources: Arena AI leaderboard, OpenRouter rankings, Anthropic: Claude Fable 5.1, Google: Gemini 3.8 Flash, Tencent: Hy4 preview