ToneBench Methodology

The benchmark measures one skill: turning a brief and a research packet into a full YouTube script, with the hook, structure, voice and technical depth of the channel's own scripts. It does not measure general intelligence, coding ability, or how a video performs once published.

1. Step 1Inputs10briefs with research packets and a shared style guide
2. Step 2Generation6,600drafts: 132 configurations × 10 scripts × 5
3. Step 3Judging4judges per draft, plus code for length and lexical slop
4. Step 4Writing Score0 to 100weighted mean of 9 metrics, averaged over drafts
5. Step 5Writing Elo8,646configuration pairs compared on each script, with 95% intervals
6. Step 6Validation8checks recomputed with each build

Judges read each script's reference version next to the draft. The models never see it.

What Changed in the October 2026 Rerun

In October 2026 the benchmark was run again from scratch. Every draft on the current board was generated in this rerun and scored by all four judges.

• Judge panel. Claude Opus 5.5 at medium effort, GPT-6.1 Sol at medium reasoning and GLM 5.3 Flash replaced the three LLM judges of the previous board. Jev remained the typed fourth judge.
• Rubric. The benchmark author wrote a new judging rubric, identical for every script and every model. Jev's quality levels were rewritten from it.
• Model list. Models whose route was retired or due to retire were removed, as were Llama 3.1 8B and Cohere's Command R, R+ and R7B. Six models that a newer generation replaced keep two effort settings: the provider default and their highest. A route is the provider and access path a model is called through, such as a provider's API, OpenRouter or a subscription command-line tool.
• Reference scripts. They remain the standard every draft is read against, but they no longer receive a score.
• Closed-book generation. Since October 11, 2026, every contestant writes without tools. Section 03 describes the change and which drafts were regenerated.

Scores from before the rerun come from different drafts, judges and rubric, and are not comparable with this board.

Each task is a real script prepared for the "What's AI" YouTube channel. Half of them come from published videos and half have not been published. Which is which is not disclosed, and the site names scripts only by number and type. Each task has three parts: a brief describing the video, a research packet with its sources, and the finished reference script. Models receive the brief and the research packet. Judges also read the reference script.

Every configuration is tested on the same 10 real scripts. Production briefs carry the constraints of an actual video, such as a target length, one call to action and a story that has to land in a set order, and they are harder to overfit than synthetic prompts. The 10 scripts cover different formats on purpose, because a model that handles a tight explainer can still fail on a personal story or a long technical walkthrough. Reference lengths run from 885 to 4,260 spoken words.

Reference Script Review

Each reference is the final version of the channel's script for that video. Judges read it next to every draft, and the rubric uses it as the voice standard; the other metrics follow their own criteria, and the research packet settles facts. The reference scripts receive no board score.

Parts of the drafting were AI-assisted; the final versions are mostly human-written and edited. Before a script became a reference, six members of the Towards AI team reviewed it: two AI engineers, a professional writer, two YouTube reviewers and a fact checker. The content lead then made a final editing pass. The team worked on each script together without scoring it separately, so no inter-rater reliability figure exists for the references.

Every configuration writes from the same prompt under the same closed-book rule. Only the model, its reasoning setting and the route that serves it differ.

Switch to Closed-Book Generation, October 2026

From August 13 to October 11, 2026, generation was open-capability: a model could use whatever its own environment offered, and access was uneven. Web search was offered on the xAI, Google and Codex routes, and 86 of 400 Grok drafts used it. Claude Code contestants were almost always denied it: 4 of 1,450 drafts recorded tool use. The other API routes offered no tools. Half of the reference scripts are published online and half are unpublished, so web access risked drafts copying a published reference. Eight drafts copied wording from published references.

Since October 11, 2026, every route is closed-book. All drafts of the 44 configurations whose route offered tools (8 xAI, 14 Google, 22 Codex) were regenerated, together with those 4 Claude Code drafts: 2,204 of the 6,350 drafts then on the board. The other 4,146 drafts recorded no tool use and were kept. All four judges scored every regenerated draft. The prompt text did not change.

Each draft receives 9 metric scores from 0 to 100, combined into one Writing Score with the editorial weights below. Four judges score 7 of the metrics; code scores length; the slop score combines a judged half and a computed half. The exact rubric wording is kept private, and the descriptions below state what each metric rewards.

Why These Weights

The weights are an editorial decision by the Towards AI team: how much each criterion matters for an AI education video that is useful and that viewers finish. Voice and substance carry the most weight; slop and visual cues carry the least. Nothing in the score predicts click-through, which depends on the title, the thumbnail, the audience and distribution, none of which is in the script. The weights may change as human feedback, retention data or editorial evidence accumulates. Section 07 reports how much the ranking depends on them.

Judge Panel

Four off-the-shelf judges score every draft: three LLM judges from three labs, one each from Anthropic, OpenAI and Z.ai, and Jev, a typed judge from TypeSafe. A typed judge is a decision model that answers fixed multiple-choice questions and writes no text. None of the judges is fine-tuned. Each judge's score for a metric counts equally: the published score is the plain average of the four judgments.

Evaluation Procedure

• Inputs. Each judge reads the brief, the research packet, the reference script and the candidate draft.
• One call per draft. A single call per judge returns all 8 judged scores: the 7 judged metrics and the judged half of the slop score.
• Blind and fixed. Judges never see which model wrote a draft. The three LLM judges use one fixed rubric prompt, the same for every script and model.
• Evidence first. Before scoring a metric, an LLM judge must quote short evidence from the draft. The quotes are stored with every run, so any score can be audited.
• Typed questions. Jev writes no text. It answers each metric as a separate question by choosing among five described quality levels, and stores a probability for each level.
• Length in code. No judge sees the word count. Length adherence and the computed half of the slop score come from code.

Calibration and Bias Controls

Jev's raw score on a metric is its expected quality level, from 0 to 4, scaled to 0 to 100. A per-metric linear map then places it on the LLM judges' scale by matching the mean and spread of the three LLM judges' average. The map was fitted once, on October 8, 2026, on all 6,350 drafts of the 127-configuration rerun cohort after the three LLM judges had scored them, and then frozen. Adding a model or a script does not refit it. The map sets the scale only: within a metric it never changes the order of drafts.

Multiple judge families are used because of an earlier measurement: when the same stored drafts were re-judged by judges from different families, every family scored its own models highest, by different amounts. With judges from four makers, no model is scored by its own family alone. This reduces the bias; it does not prove the judging is unbiased. The judge analysis page measures this own-lab preference for each judge on the current panel, and how much of it reaches the Writing Score.

How the Writing Elo Is Built

• Comparisons. On each script, every pair of configurations is compared (8,646 pairs per script). The higher Writing Score on that script wins. No separate side-by-side judgment enters this step.
• Draws. A win counts only when the gap exceeds what the two configurations' draft-to-draft variation explains: a two-sided 95% Welch t threshold on their 5 draft scores, with gaps of 1 point or less never counting. Otherwise the pair is a draw. A draw means the gap did not clear that threshold on that script; it does not show the two are equal.
• Ratings. Every rating starts at 1500. The K-factor of 16 limits how far a single game can move a rating, and expected scores follow the standard logistic curve on a 400-point scale.
• Game order. Ratings are averaged across 25 shuffled game orders of 4 passes each, so the order of games does not tilt the result.
• Intervals. Each Writing Elo has a 95% interval from a bootstrap: every configuration's draft scores on each script are resampled with replacement and the whole ranking is recomputed 200 times.

Intervals far apart are the strongest ordering evidence the board offers, though still short of proof. Heavily overlapping intervals mean the board does not settle the order of that pair; they do not establish a tie.

The checks below test whether the ranking depends on one judge, one script, the weights, draft length or the Elo procedure itself. They are computed with the board and stored in its published data. They can reveal a ranking that hinges on one factor; they cannot guarantee that no bias remains. Rank agreement is reported as a Spearman correlation (ρ), where 1 means two rankings put every configuration in the same order.

Judge Agreement

The judges differ far more in how generous they are than in which configurations they rank higher. Averaging prevents any single judge's level of generosity from setting the scale. The judge analysis page reports agreement for each pair of judges and strictness for each judge and metric.

Cost and time do not affect any score. Both are logged on every run and shown on each model's scorecard; the thinking-level page plots them against the score for every reasoning-effort setting.

Complete-Run Requirement

A configuration needs judged drafts for all 10 scripts to hold a current rank. When a configuration cannot complete a script on its exact route, because the route disappears or the model repeatedly fails to return a complete script under the frozen settings, the failure and its reason are recorded. Missing cells are never filled with a zero or an estimate, and no other model is substituted. The configuration leaves the current ranking and its archived scorecard stays visible with the reason. A configuration whose route is retired also keeps its scorecard as history without a current rank. No configuration is affected on the current board.

Full Judge Panel Requirement

Every draft on the board needs all four judges, so that no configuration scored by three judges sits next to configurations scored by four. Before any generation or judging starts, each judge is called through its own route and settings to confirm that the model answering is exactly the pinned one. If a judge cannot run, new work stops until that exact judge returns or is replaced, and a replacement re-scores every draft before the board is republished in one batch.

Scoring Versions and Ranking History

Any change to a judge, the rubric, the weights, the generation prompt, the generation conditions or the style guide starts a new scoring version, and the whole board is recomputed in one batch so that old and new scoring never mix. The October 2026 rerun started the current version; earlier boards are kept as history and are not compared with it. The Writing Elo is relative to the configurations present, so adding a model can move existing ratings; the Writing Score does not depend on the other configurations. Section 10 lists the identifiers of the current version.

Treat the ranking as directional. It is not statistically definitive, especially while the reference set is this small. It compares scripts against Towards AI's editorial criteria for educational videos and does not cover every dimension of content performance.

What the Benchmark Does Not Measure

• General ability. The benchmark covers long-form scripts for AI education videos in one editorial voice. General intelligence and coding are out of scope.
• Audience outcomes. Views and click-through are not scored; they depend on titles, thumbnails, audience and distribution.
• Unbiased judging. Judges from four makers dilute any single judge's preferences. The judge analysis measures how much of each preference still reaches the board, and the halo check in section 07, which tests whether one overall impression of a draft lifts all of its judged metrics together, shows that some correlation between metrics remains.
• Absolute Elo over time. The Writing Elo is relative to this set of configurations; the Writing Score is the steadier figure over time.
• Performance on other scripts. 10 scripts is a small set, and the intervals hold those scripts fixed.
• Actual spend. Cost is a list-price comparison.
• Human preference. The board reflects automated judges only, and the human arena is closed.

Reference Exposure

A model that has already seen a reference script can recall it instead of writing. New scripts are normally benchmarked before their video goes out. Non-public references are withheld, drafts are written closed-book with no web search, reference titles and video links are kept off the site, and the benchmark data carries a unique canary string so that training-corpus builders can filter it out. These measures lower the risk of exposure without making it impossible.

Available Evidence

• Every stored judgment keeps the LLM judge's evidence quotes or Jev's level probabilities.
• The board data, including every figure in section 07, is published as leaderboard.json and recomputed at each build.
• Every score is tied to one scoring version, listed below.