LLM Judge Panels in Research: Nine Models, Two Votes

An LLM judge panel looks like independent checking, but models make the same mistakes. What the 2026 evidence shows and the stop rule that sends work to peop...

Eight hundred open ends from a brand tracker have been coded by a model, and somebody has to sign off on the codes before the tables go out. The team does what now feels responsible. It sends the same coded file to five other models, asks each one whether every code is right, and keeps the majority verdict. The methodology note says the coding was checked by an LLM judge panel. That sentence sounds like five independent reviews. The research published in 2025 and 2026 says it is closer to one or two.

Key Takeaways

• A study of more than 350 language models found that when two models both get an item wrong, they give the same wrong answer 60 percent of the time on one leaderboard dataset.
• A panel of 9 frontier judges from 7 model families carried about as much information as 2 independent votes, and the best single judge matched or beat the panel.
• Two open judges repeated their own scores 92 to 97 percent of the time while correlating only 0.28 to 0.34 with the scores of nine human raters.
• Exact agreement overstates judge quality by 33 to 41 points against kappa, so the agreement statistic you report decides how good the check looks.
• The AAPOR task force report of May 2026 advises against relying only on language models to evaluate survey work done with language models.

What a judge panel is supposed to buy you

The logic of a panel is borrowed from human review. Two coders who disagree reveal an ambiguous verbatim. Three coders who agree are harder to dismiss than one. The arithmetic behind that intuition assumes their mistakes are independent, so each extra reviewer removes a share of the remaining error.

An LLM judge panel is a set of language models, usually from different providers, each asked to score or verify the same output, with the verdicts combined by vote, average or another rule. It is used where no answer key exists, which describes most coding and summarization work in research.

Correlated error is the tendency of different models to fail on the same items and in the same direction. When errors correlate, adding judges adds confidence faster than it adds information.

Kappa is an agreement statistic that discounts the agreement two raters would reach by chance alone. It is the measure survey methodologists use when a model's codes are compared with human codes, as the AAPOR task force report recommends in section 4.1.5.

Hold on to the independence assumption. Everything below is about what happens when it fails.

How the vote gets used on a coding project today

PERSONAL EXPERIENCE I have seen this arrangement assembled with good intentions in more than one research operation. It usually runs like this.

1. One model codes the open ends against the codeframe.
2. The coded file goes to several other models with the same instruction to confirm or reject each code.
3. Each row gets a tally, and rows with a majority in favor are marked verified.
4. Rows that split go to an analyst, if there is time before delivery.
5. The deliverable reports the share of rows the panel verified.

Steps one to four are reasonable engineering. Step five is where the method quietly overclaims. A verified share reads like an accuracy figure, and nobody in the chain compared the panel with a person.

The honest version of step five would say how often the panel agreed with a human coder on a sample. That number usually does not exist, because the panel was built to avoid needing it.

Different models make the same mistakes

The first crack in the independence assumption came from a large comparison. Correlated Errors in Large Language Models, accepted at ICML 2025, evaluated more than 350 models on two public leaderboards and on a resume screening task.

On one leaderboard dataset, when two models both erred on an item, they chose the same wrong answer 60 percent of the time. Shared architecture and shared provider explained part of that. The finding that matters for research teams is the other part. Larger and more accurate models showed highly correlated errors even when architecture and provider were different.

Read that against the usual advice to diversify. Picking models from different companies does reduce one kind of shared error. It does not return you to independent reviewers, because the strongest models converge on the same answers, including the same wrong ones.

The authors trace consequences into two settings, model as judge and hiring. In the hiring case they describe the pattern known as algorithmic monoculture: many decision makers using similar systems end up rejecting the same people. A codeframe checked by similar systems ends up missing the same verbatims.

Nine judges, about two effective votes

A second paper measured the panel directly. Nine Judges, Two Effective Votes, posted in May 2026, built a panel of 9 frontier models from 7 families and tested it on three natural language inference datasets, each item carrying 100 human annotations.

The panel behaved like roughly 2 independent votes. Its accuracy sat 8 to 22 percentage points below what nine truly independent judges would reach. The best single judge matched or outperformed the whole panel.

The authors tried the obvious rescues: different prompts, temperatures, step by step reasoning, preference tasks, smarter aggregation. The best aggregation methods closed at most 11 percent of the gap. By their estimate, about three quarters of the panel's nominal independence is lost because the models make the same mistakes on the same items.

This is the paper that gives the article its title, and the result is easy to misread in both directions. It does not say panels are useless. It says a panel of nine is counted like nine reviewers and performs like two, and that a better aggregation formula will not fix a problem that lives in the judges.

A consistent judge is not a valid judge

A judge can fail in a second way that looks even more like success. It can give the same answer every time and still be wrong in a stable, repeatable way.

When Consistency Does Not Mean Reliability, posted on 12 September 2026, tested two open models run locally as judges. They scored 300 responses to 100 questions across five domains, and nine human annotators rated every response. The two judges repeated their own verdicts 97.3 and 92.3 percent of the time. Their correlation with the human scores was 0.275 and 0.340.

Those are local open models, smaller than the ones most agencies call through an interface, so the specific figures do not travel to every judge. The shape of the result does. Self consistency tells you the instrument is stable. It says nothing about whether the instrument is pointed at the right thing.

More judges, the same mistakes

Three panels, each with its own unit. Panel one, from Correlated Errors in Large Language Models, presented at ICML 2025 with more than 350 models: on one leaderboard dataset, when two models both get an item wrong, they give the same wrong answer 60 percent of the time. Panel two, from Nine Judges, Two Effective Votes, posted on arXiv in May 2026: a panel of 9 frontier judges from 7 model families carried about 2 independent votes of information, tested on three language inference datasets with 100 human annotations per item, and the best single judge matched or beat the panel. Panel three, from When Consistency Does Not Mean Reliability, posted on arXiv in September 2026: two open judges repeated their own verdicts 97.3 and 92.3 percent of the time, while their correlation with the scores of nine human annotators was 0.275 and 0.340 on a scale from 0 to 1, over 300 responses to 100 questions. A closing line says that agreement between models is not a reference, and people are. Every number belongs to the source named beside it and none of them is a Cassi.ai measurement.

The bars that look reassuring measure models against models. The only bars that measure against people are the short orange ones.

A larger audit reached the same place from the other side. Reliability without Validity, from June 2026, ran 21 judges from nine providers through 118 runs and about 541,000 judgments on three benchmarks. The gap between exact agreement and kappa was universal, 33 to 41 percentage points on one of those benchmarks. Judge rankings also moved by up to 14 positions from one benchmark to another.

The practical reading is blunt. A report that says the panel agreed with human coders on most rows, measured by exact match, can be describing a judge whose agreement beyond chance is modest. Which statistic you choose decides how good the check looks.

A third result explains where the gap comes from. A study of model judges without human grounding, first posted in March 2025 and revised in August 2026, built a business and finance benchmark with 160 questions and 1,200 responses graded by experts. Without a reference answer, model judges agreed with the experts mainly on the questions the judges could answer correctly themselves. On the hard items, where a check is most needed, the judge has nothing to check against.

What this changes in what we already wrote

UNIQUE INSIGHT Two earlier articles on this blog set out rules that still hold, and this evidence sharpens them rather than overturning them.

In one model, one blind spot we argued that a single model applies a systematic tilt, and that running several models in parallel surfaces disagreement a researcher can rule on. That remains true. Disagreement between models is real information about a verbatim, and it tells the analyst where to look. The new evidence is about the other half of the output. Agreement between models carries less information than it appears to, because the models share failure modes.

In AI agent evaluation without an answer key we wrote that a judge from a different maker removes self preference and leaves every other bias in place. The papers above name one of those remaining biases precisely. Models from different makers still make the same mistakes on the same items. Diversifying providers is worth doing, and it does not turn a panel into independent evidence.

So the division of labor is this. Models are good at pointing at the rows that deserve a second look. Models are poor at certifying the rows that did not raise a flag. The second job needs a reference that did not come from a model.

What the survey research association now says

The American Association for Public Opinion Research published its task force report on AI in survey research in May 2026. Section 4.1.5 of that report is titled with a direct instruction not to rely solely on language models to evaluate work mediated by language models. The authors warn against evaluation built on models all the way down, where the same class of system produces the output and then serves as the main judge of its quality.

The same section does not ban model judges. For well defined tasks with clear ground truth, such as adherence to a fixed coding rubric, it treats them as useful under a layer of human supervision. It then describes the check that should sit underneath. Compare the model's codes with human codes on a shared subset of responses, using an agreement measure such as kappa, and set the acceptable thresholds before the tool goes into production.

The report also asks, in section 4.1.4, for documentation of model selection, parameters, prompt design, validation steps and the human oversight protocol. That is the paper trail an auditor or a client will ask for. We covered the agency side of that obligation in AI policy for research agencies.

What still holds: the trail, the comparison, the stop rule

None of this argues for going back to one model and trusting it. The parallel run is still worth running, for three reasons that survive the evidence.

The first is the execution trail. An execution trail is the record of every model call behind a deliverable: provider, model identifier, a hash of the exact prompt, input and output tokens, latency, and any failure or retry. The hash lets you prove which instruction produced which code without storing sensitive text. The OpenTelemetry conventions for generative AI already standardize much of this record. By default they leave out prompt content, because it can hold sensitive data.

The second is candidate comparison. Parallel outputs let an analyst see where models diverge, which is the signal the earlier articles defended. Divergence is cheap to compute and it concentrates human attention on the rows most likely to be wrong.

The third is a stop rule. A stop rule is a condition written before fieldwork that decides when model output is accepted, when it is rerun, and when it goes to a person. Here is the procedure we recommend, in order.

1. Code a calibration sample by hand before the model runs, with two human coders and their agreement recorded.
2. Run the generator and the judges, and log every call in the execution trail.
3. Measure each judge against the human sample by kappa, never by exact agreement alone.
4. Set the acceptance threshold for that task in writing, before production, as the AAPOR report asks.
5. Send every row where the models split to a person, plus a random sample of rows where they agreed.
6. Stop the run and escalate the whole task if a judge falls below its threshold on the random sample.
7. Record in the methodology note which models ran, what the human sample was, and the kappa each judge reached.

A stop rule for a model panel

A decision flow in four steps, written before production, each with an exit to the right. Step one, code a sample by hand first, with two human coders and their own agreement recorded. If there is no human sample, the panel has nothing to be valid against and the work should not start. Step two, measure each judge by kappa against the human sample, never by exact agreement alone. If a judge falls below the threshold written in advance, remove it from the panel for this task. Step three, read where the models split: every row with a split verdict goes to a person. If split rows are left unread, the disagreement signal was wasted. Step four, sample the rows the models agreed on: a random share of agreed rows goes to a person. If that sample fails, stop the run and escalate the whole task. A closing band says to record every call, meaning provider, model, prompt hash, tokens, latency, failures, and each judge’s kappa against people. The figure contains no measured numbers.

Step four is the only one that can catch a mistake every model made together. Without it, a shared blind spot passes as verified.

Step five is the one that answers correlated error. The random sample of agreed rows is the only part of the design that can catch a mistake every model made together. Without it, a shared blind spot passes through as verified.

A stop rule you can hand to a project manager

UNIQUE INSIGHT The rule has to fit on one page, or it will not survive a deadline. This is the decision table we use to explain it.

Sources for the rows: Nine Judges, Two Effective Votes, Reliability without Validity and the AAPOR task force report. The mapping onto coding work is ours.

The last row deserves emphasis. The AAPOR report states, in section 4.1.2, that strong performance in one application should not be assumed to carry over to a new task. A panel calibrated on a beverage tracker in one language is uncalibrated on a banking tracker in another.

Who answers for the verified share

PERSONAL EXPERIENCE The failure I worry about is not dramatic. It is a client challenge six months after delivery.

A regional manager disputes a code, the one that moved a satisfaction driver up the chart. The agency opens the file and sees that five models agreed on that row. Nobody can say whether a person ever read a sample of agreed rows, which models ran, or what their agreement with human coders was.

At that moment a panel without a trail is weaker than a single careful coder, because the coder can at least explain the decision. The trail and the human sample turn the panel back into a method someone can defend. The evidence chain for the codes themselves is the subject of our article on qualitative coding.

Where Cassi.ai comes in

@Cassi.ai is a software engineering company specialized in the pains of market research, innovation and insights, founded by Cassiano Albuquerque. We work with research agencies and corporate insights teams.

Synthetic Lab is a Cassi.ai product that turns the research already done into synthetic personas of the target, for synthetic focus groups and quantitative studies with quotas.

Our multi-model work started with a benchmark of frontier models across sentiment, rationality, cultural openness and ethical foundation, presented publicly at IIEX LATAM in 2025 and summarized in one model, one blind spot. Its finding was that models diverge systematically rather than randomly. That finding is why we run models in parallel. The evidence in this article is why we never let the parallel run certify itself.

The architecture we describe for clients is a multi-model consensus, synthesized by agents and audited by human researchers. In that sentence, audited carries the weight. Disagreement goes to a researcher. Agreement gets sampled. The prompt, the model versions and the divergence log stay attached to the study. The discipline we hold ourselves to is to report agreement with human coders by kappa, and never a verified share on its own. The rest of our work sits in the cassiai.com portfolio.

FAQ

Does using multiple LLMs as judges improve accuracy?

Less than the count suggests. In Nine Judges, Two Effective Votes, 9 judges from 7 families carried about 2 independent votes of information, and the best single judge matched or beat the panel. A panel helps most as a way to find disagreement.

Why do different LLMs make the same mistakes?

They are trained on overlapping data with similar methods, and the strongest models converge. Correlated Errors in Large Language Models found highly correlated errors among larger, more accurate models even across different architectures and providers.

Is LLM as a judge reliable without human labels?

Stable, often. Valid, not demonstrably. Judges in one 2026 study repeated their own scores more than 92 percent of the time while correlating weakly with human raters. Without a human reference, stability is the only thing you have measured.

How do you measure agreement between an LLM judge and human coders?

Code a shared sample by hand and compare with kappa, which removes chance agreement. The AAPOR task force report names kappa for this purpose. Exact agreement alone overstated judge quality by 33 to 41 points in Reliability without Validity.

When should model disagreement go to a human reviewer?

Always, for the rows where models split. Add a random sample of the rows where they agreed, because that sample is the only check on mistakes the models share. Stop and escalate the task if a judge falls below the kappa threshold set before production.

Nine models agreeing on a code is one piece of evidence that looks like nine. The trail, the human sample and the stop rule are what make the tenth reader, the person, count.

If your team reports a verified share from a model panel today, take fifty of the verified rows at random and have a coder read them cold. The disagreements you find are the part of the error the panel was never able to see.

About the author

Cassiano Albuquerque is the founder of @Cassi.ai. He has spent twenty years in market research operations, from fieldwork and tabulation to the reports that reach steering committees, and now builds AI systems for research teams. To discuss how a multi-model check on your coding can be calibrated against human coders, contact him on LinkedIn or see the cassiai.com portfolio.

Published by Cassi.ai. Read the full article at https://www.cassiai.com/blog/nine-models-two-votes.