The LLM Leaderboard Is Measuring Agent Economics Now
The most important number on this week’s model leaderboard is not an Elo score. It is 126,000: the rough number of tokens in the average request sent to MiniMax M3.1 Flash through OpenRouter. That is not ordinary chatbot traffic. It looks like agents reading repositories, calling tools, revising plans, and running long enough for token economics to become an architectural concern.
OpenRouter’s trailing-week ranking puts DeepSeek V4.1 Flash 1017 first with 33.56 trillion tokens, an 801% week-over-week increase. The previous V4.1 Flash sits second at 18.00 trillion. Together, those two DeepSeek entries handled 51.56 trillion tokens—more than the next four models combined. MiniMax M3.1 Flash, offered as a free variant, debuted at number three with 13.68 trillion tokens across 108.3 million requests.
Those are startling figures, but they are easy to misread. OpenRouter is measuring prompt, completion, and native reasoning tokens routed through one platform in a trailing UTC window. It is not measuring unique users, revenue, total market share, accuracy, or developer preference in isolation. Private requests are excluded, and free and paid variants are separate entries. A verbose model receives more leaderboard credit than a concise one for completing the same job.
That limitation is not a reason to ignore the ranking. It is the reason to read it correctly. Quality benchmarks tell teams what a model can do under a controlled test. Router traffic shows what developers are willing to put into actual loops once price, latency, availability, and integration friction join the evaluation. The current board says the low-cost workhorse tier has stopped being a side market.
Free inference is buying a place inside the agent stack
MiniMax says M3.1 Flash is the public successor to its anonymous Space Bunny preview, which processed more than 200 trillion tokens. The company claims output above 200 tokens per second, a P50 above 170 TPS, nearly three times the online inference speed of its predecessor, and a 70% reduction in agent-task cost. Direct API pricing is $0.15 per million input tokens, $0.60 per million output tokens, and $0.015 per million cached-input tokens.
The vendor also reports 73.7% on DeepSWE v1.1, 61.7 on OSWorld V2, 29.6 on Agents’ Last Exam, 76.2 on BankerToolBench, and 82% on CyberGym. Those results are useful as test candidates, not purchasing conclusions. They come from the vendor, span different task definitions, and cannot be collapsed into a single notion of “best.” The relevant question for an engineering team is narrower: does this model finish our jobs reliably enough that its price advantage survives retries and review?
The free label makes that question urgent. Free inference is customer acquisition for agent workloads. A chat user can switch products in seconds; an agent platform becomes embedded in prompts, routing policies, tool schemas, evaluation suites, and failure handling. Subsidizing the first few trillion tokens is a rational way to earn that integration. The bill can change later, but the operational dependency remains.
That creates a less obvious risk than conventional vendor lock-in. Teams may believe they selected a model because it was technically superior when they actually selected it under a temporary price regime. If the free endpoint disappears, becomes rate-limited, or changes its data terms, the migration cost arrives precisely when the subsidy ends. Treat free model access as a trial with an unknown expiration date, not as a permanent unit-economics assumption.
Tokens per second can still produce a slow task
The Hacker News response to M3.1 Flash was appropriately skeptical. The launch drew 517 points, but practitioners kept returning to wall-clock completion time. Plotly cofounder Chris Parmer reported that the model answered 41 of 43 questions correctly on an internal data-analytics exam, tying Claude Opus 5.5 and Gemini Argon at roughly one-third of their price. He also said it was the slowest model tested because it generated so many tokens.
That distinction matters. Throughput measures how quickly a provider emits tokens after generation begins. Task latency includes planning, tool calls, round trips, retries, and the total number of tokens the model decides to produce. A model running at 200 tokens per second can lose to one running at 100 if it generates three times as much text or takes an unnecessary detour through the tool loop.
OpenRouter’s ranking makes the same trap visible from another angle. Step 5 Preview climbed three places to number 12 even while its weekly usage fell 68%. Its neighbors simply fell faster. GPT-6.1 Sol rose two spots to number 15 on 45% growth, while Muse Spark 1.3 Contributor dropped three to number 17. Rank is relative; movement is not adoption, and token volume is not outcome quality.
Even the apparent contest at number six is mostly noise. GPT-6 Luna routed 8.35 trillion tokens and Claude Opus 5.5 routed 8.27 trillion, a difference of less than 90 billion tokens—effectively a rounding error at this scale. More interestingly, Opus still grew 34%. Cheap models are expanding without eliminating premium ones, which suggests that teams are segmenting work rather than choosing a single universal default.
Measure cost per accepted result
The practical metric is cost per accepted outcome. It includes model charges, failed attempts, human review, recovery calls, and the opportunity cost of waiting. A $0.60-per-million-output-token model that loops, over-explains, or silently corrupts a file can be more expensive than a premium model. A low-cost model that handles routine repository searches, test generation, or data extraction cleanly can reduce spend without touching the hard cases.
Teams considering M3.1 Flash or DeepSeek V4.1 Flash should run a shadow evaluation on representative traces before changing production routing. Sample a few hundred real tasks, remove sensitive data, and give every candidate the same tool permissions and budget caps. Record accepted-task rate, p50 and p95 wall time, input, output, cached and reasoning tokens, tool-call failures, retries, and minutes of human rework. Report results by task class; an aggregate score can hide a model that is excellent at retrieval and unreliable at code modification.
Then build routing around those classes. Use the inexpensive model for bounded, reversible work. Reserve the stronger model for planning, review, ambiguous changes, and recovery. Add explicit fallbacks, maximum loop counts, and spend ceilings before allowing any newly launched model to own unattended production jobs. Pin model versions where the provider allows it, because a convenient alias can silently turn an evaluation into a moving target.
Free endpoints deserve an additional security review. Confirm retention, training, regional routing, and subprocessors before sending proprietary code or customer data. If those terms are unclear, synthetic or scrubbed evaluation traces are the limit—not a loophole to hope the word “free” covers.
Arena would normally provide the complementary judged-quality view, but its Text and WebDev tables could not be verified in the latest research pass: the dedicated extractor returned no rows and the fallback was blocked by Cloudflare. Publishing invented continuity would be worse than publishing no Arena movement at all. The honest story is therefore narrower and more useful: this week’s OpenRouter data captures workload gravity, not model intelligence.
The leaderboard’s winner is not necessarily DeepSeek, MiniMax, or whichever name moves up three places next week. The winner will be the model that produces the lowest cost per accepted result after verbosity, latency, retries, review, and operational risk are counted. Trillions of tokens show where developers are experimenting. Only disciplined evaluation shows where they should stay.
Sources: OpenRouter Rankings, OpenRouter weekly rankings API, MiniMax M3.1 Flash announcement, Hacker News discussion