LLM Judges Can Infect Each Other With Bias — Measure the Network Before You Trust the Score

LLM judges were supposed to make evaluation cheaper. Then they became part of the system being evaluated.

That is the uncomfortable premise behind Contagion Networks: Evaluator Bias Propagation in Multi-Agent LLM Systems. The paper studies what happens when model agents evaluate each other and update behavior from those evaluations. The answer is not “wisdom of crowds.” It is closer to “bias with a network topology.”

This matters because LLM-as-judge has quietly become the duct tape holding AI evaluation together. Teams use model judges to grade support answers, code patches, RAG faithfulness, red-team traces, synthetic data, agent trajectories, and benchmark submissions. When one judge feels too brittle, the common move is to add more agents and call it a committee. That sounds robust. It may just give the same bias more routes to propagate.

The paper introduces a Cross-Agent Contagion Matrix, written as Γ_N, to measure how evaluator preferences move through an N-agent system. The controlled experiment uses three DeepSeek-chat / deepseek-v4-flash agents with different evaluator-bias profiles: structured, balanced, and evidence-based. Same underlying model family, different evaluator personalities. That constraint is useful because it isolates the topology problem from the easy explanation that “different models behave differently.”

Consensus is not the same thing as independence

The pairwise contagion results are positive across all six off-diagonal paths. Mean gamma values land roughly between 0.143 and 0.304, with the abstract reporting a range around 0.157 to 0.352. The strongest outward bias comes from the evidence-biased evaluator, where the C → B path reaches 0.304 ± 0.068. In plain terms: one evaluator’s scoring preference can measurably alter another agent’s behavior.

That alone should change how teams think about multi-agent review. If a critic that rewards exhaustive answers causes a generator to become verbose, you do not have a better writer; you have evaluator-induced verbosity drift. If a security reviewer overweights scary language, a triage system can become paranoid. If a balanced judge absorbs the preferences of a more forceful evaluator, the committee’s apparent moderation is cosmetic.

The topology results are the real warning. All chain-link contagion coefficients are below 1.0, so the homogeneous DeepSeek setup attenuates under a chain topology. After three hops, the cumulative factor drops to β₃ = 0.0055. That sounds safe until you look at dense graphs. The full pairwise matrix has a spectral radius around 1.402-1.404, implying the same agents can look stable in one wiring diagram and riskier in another.

This is the part product teams routinely skip. They document the model, maybe the rubric, sometimes the prompt, almost never the evaluator graph. But in multi-agent systems, the graph is part of the model behavior. Who reviews whom, who gets to update, whether judgments are visible to other agents, whether critiques are sequential or simultaneous — these are not implementation details. They are bias propagation controls.

Three judges help, but only if diversity is real

The paper reports that increasing evaluator committee size from k=1 to k=3 reduces effective contagion from 0.264 to 0.073, a 72.4% reduction. That is the number everyone will want to quote. Quote the caveat too: adding agents is not magic if the agents share blind spots, rubrics, training artifacts, or update paths.

The comparison to prior cross-model MM-EPC results makes this more interesting. Homogeneous-model contagion coefficients are reported as 3-5× weaker than cross-model values around 0.85-1.3. That cuts against the lazy version of “just use diverse models.” Cross-model committees can be better, but they can also introduce stronger preference transmission if one model’s evaluation style dominates another’s adaptation. Diversity must be measured, not asserted in a YAML file.

For practitioners, the checklist is straightforward. If your benchmark or product loop uses LLM judges, publish or at least internally track the judge model, prompt, rubric, sampling settings, topology, disagreement rate, and known bias probes. Run ablations where evaluators are isolated. Measure whether a generator’s outputs change after exposure to a judge. Track verbosity, position bias, self-preference, severity drift, evidence preference, and calibration against human-reviewed gold sets. If you cannot explain why a committee’s consensus is independent evidence rather than contagious preference, do not present it as independent evidence.

There is also a cost angle. The experiment used 840 DeepSeek-chat API calls, which is small enough that teams can run this kind of diagnostic routinely. That is the right framing: evaluator-contagion testing should be part of eval infrastructure, not a one-off research stunt. If a company can afford weekly leaderboard refreshes, it can afford to measure whether the judges are teaching each other bad habits.

The paper is not saying LLM judges are useless. They are often the only scalable way to evaluate open-ended model behavior. The mistake is treating them like neutral instruments. A model judge is another learned system with preferences, quirks, incentives, and failure modes. Put several of them in a network and you have a social system made of stochastic parrots with API keys. That system can be useful. It can also converge on nonsense with a very professional confidence interval.

The editorial takeaway is simple: evaluation cards need a topology section. The next time a benchmark says “scored by a panel of LLM judges,” ask which judges, wired how, with what contagion matrix. If that sounds too fussy, remember that the whole point of evaluation is to distinguish signal from shared delusion. Multi-agent judging is not automatically safer than one judge. It is a network, and networks propagate.

Sources: arXiv, MT-Bench / LLM-as-a-judge, AlpacaEval verbosity-bias context, MM-EPC