Python-Only Coding Benchmarks Are Lying by Omission

Python has been doing too much unpaid PR work for coding models.

For the last few years, “best coding model” has usually meant “best at Python-heavy benchmarks, plus vibes from a few repository demos.” That was convenient for leaderboard maintenance, not especially honest about software engineering. Multi-LCB, a new arXiv benchmark extending LiveCodeBench across twelve programming languages, is useful because it attacks the quiet assumption underneath most coding-agent comparisons: that Python performance is a reliable proxy for general code generation.

It is not. And the paper’s rankings make that hard to ignore.

Multi-LCB takes LiveCodeBench’s contamination-aware setup and translates the task universe beyond Python into C++, Java, Go, JavaScript, TypeScript, C#, Rust, Ruby, PHP, Kotlin, and Scala. It keeps the same basic evaluation discipline — release-date filtering, hidden tests, Pass@1 — while asking a more realistic question: if a model has to work in the languages real codebases actually contain, does the Python leaderboard still tell the truth?

The answer is “partly,” which is exactly the uncomfortable answer benchmark buyers should want.

The leaderboard changes when the codebase stops being a notebook

The paper evaluates 24 public LLMs across model families including GPT-OSS, Qwen3, DeepSeek, OlympicCoder, OpenReasoning, OpenCoder, Devstral, and Seed-Coder. The authors ran the experiments on 16 NVIDIA H100 80 GB GPUs using vLLM or SGLang, and they focus on a February 2025 through May 2025 task slice to reduce contamination risk for newer models.

On that slice, GPT-OSS-120B Medium leads the multilingual average at 67.8% Pass@1. Qwen3-235B-A22B-Thinking-2507 follows at 64.0%, and DeepSeek-R1-0528 lands at 63.1%. If that were the whole story, this would be another ranking article with a slightly different table. The useful part is the per-language shape.

Qwen3-235B-A22B-Thinking-2507 is strongest on Python at 74.0%, C++ at 75.8%, Java at 73.9%, and PHP at 69.0%. GPT-OSS-120B Medium wins Go at 69.9%, JavaScript at 70.5%, TypeScript at 70.3%, Rust at 70.5%, Ruby at 70.2%, and Kotlin at 71.0%. DeepSeek-R1-0528 is best on Scala at 62.3% and second-best on Rust and Ruby, despite trailing Qwen on Python.

That is the whole plot. A single “coding score” compresses away exactly the information an engineering team needs. If your stack is Python and Java, Qwen looks excellent. If your stack is TypeScript, Go, Rust, and Kotlin, the answer changes. If you are maintaining a glorious enterprise sedimentary layer of Java, C#, PHP, Scala, and suspiciously load-bearing Ruby, the idea that one leaderboard position should drive procurement is mostly theater.

The authors state the point directly: “strong Python ability is not always a reliable proxy for true cross-lingual code generation competence.” Correct. Also overdue.

Routing by language beats worshipping one champion model

The practitioner implication is not “switch to the model currently first on Multi-LCB.” That would just be leaderboard-chasing with better inputs. The implication is that coding-agent control planes should route by language, task type, cost, latency, and feedback from tests.

This matters because coding agents are no longer just autocomplete with a bigger context window. They edit files, run tests, open PRs, and burn budget while doing it. In that world, using a single model because it wins a Python-heavy benchmark is like choosing one compiler flag for every service in the company because it improved the frontend build once. It might work. It is also not a strategy.

A better pattern is straightforward. Inventory the languages in your repositories. Build a small internal benchmark from real bugs, migrations, and review comments. Weight the benchmark by where engineers actually spend time, not by what is fashionable on arXiv. Then route tasks: cheaper models for boring Python edits, stronger models for languages where the benchmark says they win, escalation after test failure, and human review earlier for languages where all models are weak.

That last point is where the cost story gets interesting. Most AI-coding cost analysis treats tokens as the primary unit. Tokens matter, but failed patches also cost review time, CI time, context-switching, and trust. A model that is marginally more expensive but substantially better at Rust ownership fixes may be cheaper than a bargain model that produces plausible nonsense three times. Conversely, throwing a frontier reasoning model at every Python refactor is likely waste. Multi-language benchmarking gives teams a way to cut spend without pretending the cheapest model is always the cheapest outcome.

Vendors should feel some pressure here. If an agent product sells into professional engineering teams, “we score well on coding benchmarks” is not enough. Publish language breakdowns. Show Java, TypeScript, Go, C#, and Rust. Show where the system degrades. If the answer is “we do not know,” that is not a technical impossibility; it is a product choice.

The benchmark is better, not final

Multi-LCB still inherits some limits from the benchmark genre. Pass@1 on competitive-style programming tasks is not the same thing as landing a safe production patch in a tangled repository with flaky tests, stale documentation, and a senior engineer asking why the migration touched billing code. Translation across languages can also introduce artifacts: idiomatic Rust is not merely Python with ownership syntax, and enterprise Java problems are not always algorithm puzzles wearing braces.

But that is not a reason to dismiss it. It is a reason to use it properly. Multi-LCB is a stronger public signal than Python-only leaderboards because it exposes ranking instability across languages. It should feed routing policy, vendor evaluation, and internal benchmark design. It should not become another number copied into a slide deck with all the inconvenient columns hidden.

The editorial take is simple: if your coding-agent leaderboard is Python-only, it is not measuring software engineering. It is measuring Python benchmark fluency. That can still be useful, in the same way a unit test can be useful. Just do not mistake it for production readiness.

Sources: arXiv, Multi-LCB project repository, LiveCodeBench leaderboard, Qwen3 model card, DeepSeek-R1-0528 notes