ToneBench Judge Panel Analysis

Every draft on the board is scored once by each of the four judges, blind to the model that wrote it, and the Writing Score is their plain average. Claude Opus 5.5 is the strictest judge and GLM 5.3 Flash the most generous. Within one configuration, the judge that scores it highest and the judge that scores it lowest are 15.1 points apart on average, whichever judges those are.

The average is over all 6,350 drafts. Spread is the standard deviation of a judge's average score across the 127 configurations, with its lowest and highest configuration below it: a larger spread means the judge separates strong and weak configurations more. Jev's raw answers are mapped onto the LLM judges' average scale for each metric (calibration), so its average matches the LLM judges' average by construction and says nothing about its own generosity.

Judges differ in overall generosity, so raw averages cannot show a preference for one lab. Instead, each judge is compared with what the other LLM judges' scores predict for the same configurations. The difference, in Writing Score points, is the judge's lab preference: a positive value means the judge gives that lab's drafts more points than the other LLM judges' scores predict, a negative value fewer, and zero means no preference.

Lab Preference Values

Each value is in Writing Score points, with its 95% interval below it. Values in bold have an interval that excludes zero.

How the Values Are Computed

1. Each judge's average Writing Score is taken for each of the 127 ranked configurations, over its 10 scripts and 50 drafts: 6,350 drafts per judge.
2. The reference for an LLM judge is the average of the other two LLM judges. The reference for Jev is the average of the three LLM judges. Jev stays out of the LLM judges' reference so that its own strong preferences do not appear as preferences of the others.
3. For each lab, a second-degree polynomial that predicts the judge's score from the reference is fitted by least squares on every configuration from the other 14 labs.
4. The lab's value is the average gap between the judge's score and the curve's prediction over that lab's own configurations.
5. The 95% interval is a bootstrap: the 10 scripts, and the configurations within each lab, are resampled with replacement 1,000 times and every step is repeated. The middle 95% of the results is the interval.

How the Reference Changes the Values

An LLM judge is compared with the other two, so a preference of one of them moves the others' values for the same lab in the opposite direction. The largest case on this board: Claude Opus 5.5's OpenAI value is -2.2 points against the average of GPT-6.1 Sol and GLM 5.3 Flash, and +1.5 against GLM 5.3 Flash alone, so all of that value comes from GPT-6.1 Sol's preference for its own lab. Every value for another LLM judge's own lab carries some of this effect.

The own-lab values depend on the reference too. Each one against both other LLM judges, then against each of them alone:

• Claude Opus 5.5 for Anthropic: +1.9 against both, +3.2 against GPT-6.1 Sol alone, +0.7 against GLM 5.3 Flash alone.
• GPT-6.1 Sol for OpenAI: +3.7 against both, +3.3 against Claude Opus 5.5 alone, +4.4 against GLM 5.3 Flash alone.
• GLM 5.3 Flash for Z.ai: +0.5 against both, +0.4 against Claude Opus 5.5 alone, +0.7 against GPT-6.1 Sol alone.

Each own-lab value whose interval excludes zero stays positive against either judge alone; its size depends on which judges form the reference.

Judging is blind: no judge is told which model wrote a draft. A judge can still favor the writing style its own lab's models share. Each card shows an LLM judge's preference for its own lab, from the same calculation as the chart in section 02, and what the board would look like if that judge scored it alone.

Preference for the Judge's Own Model

Each LLM judge runs a model that also competes on the board. The own-model value applies the same curve as the own-lab value, fitted outside the judge's lab, and averages the gap over that model's configurations only. Values are in Writing Score points, with the 95% interval below each; values in bold have an interval that excludes zero.

The intervals come from a separate bootstrap of 1,000 samples that resamples the scripts, and the model's configurations among themselves, so the own-lab intervals keep their values. GLM-5.3 Flash has one configuration on the board, so its interval comes from resampling the scripts alone.

Jev has no lab of its own on the board. Its two largest preferences, Alibaba +3.7 and Google -2.6 points, are about as large as the largest self-preference above. Self-preference is therefore only one source of lab preference on this panel.

Ranks in this section are by Writing Score among the 127 configurations. The board itself ranks by Writing Elo, which follows the Writing Score closely.

The Pearson correlation (r) measures how closely two judges' scores move together: 1 means they rise and fall in step, 0 that they are unrelated. On single drafts it runs from 0.81 (GPT-6.1 Sol and Jev) to 0.94 (Claude Opus 5.5 and GPT-6.1 Sol). Averaging a configuration's 50 drafts cancels most draft-level disagreement, so the configuration figures are higher.

Each Judge's Ranking Against the Writing Score

The Spearman correlation (ρ) compares two rankings: 1 means the same order. Each judge's ranking of the configurations, by its own average, is compared with the Writing Score ranking, which includes that judge, and with the average of the other three judges, which does not. The board itself ranks by Writing Elo, so a configuration's place by Writing Score can differ from its configuration rank on the board. Jev agrees least with the others (ρ 0.909): it shares six of the Writing Score top 10, and its #1 is Claude Fable 5.1 (max effort), #5 by Writing Score.

Claude Opus 5.5 gives the lowest average on seven of the eight judged metrics, and GLM 5.3 Flash the highest on all eight. The widest gap is on Slop Score, judged half: 22.9 points between the strictest and the most generous judge. Each cell is the judge's average over all 6,350 drafts; its color shows the difference from the four-judge average for that metric. Length and the computed half of the Slop Score are measured by code and are the same for every judge, so they have no row of their own.

Jev's scores are mapped onto the LLM judges' average scale metric by metric, so its column matches the panel average by construction and says nothing about its own strictness.

The board's order comes from rubric scores. As a separate check, Claude Opus 5.5 read two anonymous drafts of the same script side by side, in both orders, without their rubric scores, and picked the better one. Each of the top 15 configurations was paired with every other top-15 configuration up to two places away on the board, on all 10 scripts.

Most of these pairs are near neighbors whose 95% Writing Elo intervals overlap, so the board does not claim an order for them. The pairs that could contradict the board are the ones it treats as settled, with intervals that do not overlap. The other two pairs split evenly between the two drafts, so they count toward neither order. The full head-to-head ranking of the top 15 has a Spearman correlation of 0.30 with the board, a noisy figure because it rests on close neighbors.

Claude Opus 5.5 is also the panel's Anthropic judge, and twelve of the top 15 configurations are Anthropic's. Its head-to-head order places the other three configurations 0.3 places higher on average than the board does, and Anthropic's twelve configurations 0.1 places lower. This check shows no lean toward Anthropic's configurations.

The Writing Score averages the four judges with equal weight. A preference of one judge therefore enters every score at about a quarter of its size, and the preferences of judges from different labs pull in different directions. To measure what remains, the board is recomputed without each judge in turn, and the average Writing Score and rank of each lab shown are compared with the published ones.

The largest preference on the panel, GPT-6.1 Sol at +3.7 points for OpenAI, changes OpenAI's average Writing Score by +1.0 points on the board. It still moves OpenAI's configurations 6.3 places on average, because ranks are crowded: among the top 20 configurations, neighboring configurations are a median 0.07 Writing Score points apart.

Effect of Each Judge by Lab

Each cell shows how many Writing Score points the judge's presence adds to the lab's configurations, relative to the rest of the board, and how many places it moves them by Writing Score. Positive values mean the judge lifts the lab.

What This Analysis Cannot Detect

Every value on this page compares a judge with the other judges. A preference that all four judges share, for example for a writing style that several labs' models produce, cancels out of these comparisons and stays in the Writing Score. Only an outside reference, such as blind human ratings, can measure it. The methodology page lists the board's other reliability checks.