ToneBench

ToneBench is a writing benchmark for one practical task: writing a YouTube script in the voice of a specific creator, Louis-François Bouchard of the What's AI channel. Every draft is scored against the creator's finished script for the same brief, so the benchmark measures how faithfully a model reproduces that voice and its craft. It does not measure general intelligence.

Louis-François Bouchard, co-founder and CTO of Towards AI and host of the What's AI YouTube channel, maintains ToneBench with the Towards AI editorial team, which reviewed the reference scripts. The about page has more on the team.

General-purpose leaderboards measure broad preference: which answer a crowd or a judge prefers on average across many tasks. They say little about whether a model can hold one specific voice, follow a strict style guide, avoid common AI phrasing and reach a target length for a real production brief.

ToneBench measures that narrower ability. Its scripts come from private production briefs instead of a public dataset, which reduces the risk that models have memorized them (see the question on private reference scripts below).

Every model writes the same set of real video scripts from the same prompt: a style guide built from the creator's voice and storytelling guides, the target length, the brief and its research packet. No model uses web search or any other tool, so the brief and the research packet are the only source. Each script is written five times, because a single draft can be unrepresentative.

A panel of four judges scores every draft blind: three LLM judges from three labs (Claude Opus 5.5 at medium effort, GPT-6.1 Sol at medium reasoning and GLM 5.3 Flash) and Jev, a typed judge from TypeSafe calibrated on the three LLM judges. The LLM judges quote the script before scoring each metric, so every grade points to evidence in the text. Jev answers each metric as a multiple-choice question among five described quality levels.

Every metric except Length Adherence and the Slop Score is the average of the judge panel. Length Adherence is computed in code from the spoken word count. The Slop Score is half judged and half computed by the EQ-Bench slop-score pipeline and a banned-phrase scan, with a small deduction for each spoken dash after the third. The nine metric scores combine with fixed weights into the Writing Score.

The Writing Score is absolute: a 0 to 100 weighted average of the nine metric scores, averaged over every draft and every reference script.

The Writing Elo is relative: for each script, every pair of models is compared on their scores, and the higher score wins unless the gap is within the two models' draft-to-draft variation, in which case the pair is a draw. The wins, losses and draws produce the rating, and resampling the draft scores gives each rating a 95% interval.

The Writing Score describes how good the writing is. The Writing Elo describes which model beats which on this board, and it can shift when models are added, so the Writing Score is the steadier number over time.

Start with the models at the top of the Writing Elo or Writing Score ranking and check whether their 95% Elo intervals separate them from their neighbors. Then look at the metrics that matter for your use in the metric rankings, such as Tone & Voice Match for a personal voice or Substance, Accuracy & Value for technical content.

Next, compare cost per script in the Quality vs. Price chart, where the price frontier shows the best score available at each price. Finally, open the scorecards and read example scripts, or tick two models in the results table to compare them across every thinking level.

Cost per script is the cost of producing one complete script at the provider's API list price, so every model is priced on the same basis. For API runs, input is what the provider billed. Models write closed-book, so no search results or other tool output are fed back as input.

Models run through a subscription (Claude Code or the Codex CLI) have no per-call bill, so their cost is rebuilt from the provider-reported output tokens for the run, reasoning included, plus the fixed prompt billed once as input. A provider route that served a model for free shows a cost of $0. An asterisk (*) marks an estimated price. Cost is a comparison figure and is not part of the Writing Score or the Writing Elo.

The price frontier is the dotted line in the Quality vs. Price chart. It connects the models that score higher than every cheaper model on the selected metric. Read from left to right, it shows how much more quality each step up in price buys.

A model below the line has an alternative that costs the same or less and scores at least as well. The frontier changes with the selected metric, script and view.

Ranks compare models. A model can run at several reasoning-effort or thinking settings, and each setting is one configuration, benchmarked on its own. A model's rank is the place of its best configuration, the one with the highest Writing Elo, among the best configurations of all models.

The leaderboard opens on the Best per model view, which shows those model ranks. All configurations ranks every configuration instead, so a model can appear several times there. Each scorecard gives both the model rank and the configuration rank.

In the All configurations view, each row is one model configuration: a specific model at a specific reasoning-effort or thinking setting, sometimes served through a specific provider route. Each configuration is benchmarked separately because the setting changes both quality and cost.

The default Best per model view keeps each model's best configuration on the selected metric (its cheapest when the table is sorted by cost), so each model appears once, and Best per lab keeps one per lab. The thinking-level page compares the settings of each model.

Each model writes every script five times, and its scores are averaged over those drafts. In the Elo comparisons, a win on a script counts only when the score gap exceeds what the two models' own draft-to-draft variation can explain (a two-sided 95% Welch t-test threshold on the draft scores), and a gap of one point or less is always a draw.

The 95% Elo interval comes from resampling the recorded draft scores and recomputing the ranking. Each model's scorecard shows how much its scores vary between drafts.

They can be, and the bias was measured. Before the panel, when a single judge scored the board, re-judging the same stored drafts with judges from other labs showed every lab's judge scoring its own models highest, by different amounts. The judged scores now come from a panel of four judges: Claude Opus 5.5 (Anthropic), GPT-6.1 Sol (OpenAI), GLM 5.3 Flash (Z.ai) and Jev (TypeSafe).

The three LLM judges also have models on the board, but every judged metric averages all four judges, so no model is scored by its own lab alone, and TypeSafe has no model on the board.

Two further checks run on top of the panel. Claude Opus 5.5 compares the top-ranked configurations with their nearest-ranked neighbors head to head, reading two anonymous drafts in both orders. The board's wins, losses and draws are also refitted with a Bradley-Terry model, a rating fit that, unlike Elo, does not depend on the order of the comparisons. The methodology page reports the current agreement figures.

The judge analysis page measures which labs each judge scores higher or lower than the other judges do, and how much of that reaches the Writing Score.

Tone & Voice Match: how closely the script matches the creator's anti-hype, first-person voice. Writing Craft & Clarity: sentence-level quality when read aloud. Substance, Accuracy & Value: accuracy, depth and grounding in the research packet. Continuity & Emotion: whether each paragraph leads to the next and each section carries a clear emotional register. YouTube Best Practices: structure and retention practices, such as an open loop that pays off and a subscribe call that fits the script. Hook Strength: whether the opening holds the viewer. Length Adherence: spoken length against the target, with overshooting penalized more than undershooting. Slop Score: freedom from overused AI phrases and patterns (higher means less slop). Visual Cue Quality: the usefulness of the [SHOW:] visual directions.

The weights appear under the results table.

To limit contamination: if the briefs, research packets and reference scripts were public, future models could train on them, and the benchmark would measure memorization instead of writing. The exact rubric wording is also withheld so that models cannot be tuned to the rubric.

Half of the reference scripts come from published What's AI videos, so their spoken version is public, and half have not been published. Which is which is not disclosed, and the site names scripts only by number and type.

New scripts are normally benchmarked before their video is published, and all benchmark data carries a canary string that asks model trainers to exclude it. These measures reduce the risk of exposure without eliminating it.

New models are added after release once a configuration has five drafts for every reference script and every judge on the panel has scored them.

When the rubric, a judge or the weights change, the whole board is re-judged in one batch, and every result records its evaluation version.

The October 2026 rerun went further: every model on the board wrote every script from scratch, a new panel of four judges scored every draft from a revised judge prompt, and models that their provider has retired or scheduled for shutdown were dropped. Older models with no announced retirement, such as GPT-4o, Gemini 2.5 Pro, Claude Haiku 4.5 and Llama 4 Maverick, stay on the board. Scores from before the rerun are not comparable with current ones.

Not necessarily. Each model's 95% Elo interval shows how much its rating moves when the recorded draft scores are resampled and the ranking is recalculated. When two intervals sit far apart, the order is well supported on this board. When they overlap, the pair is too close to call.

Neither case is proof: comparing two separate intervals does not test the gap between them directly, and overlap does not establish a tie. The reference scripts stay fixed, so the intervals also say nothing about how the order would change on new scripts. Read the metric breakdowns and example scripts alongside the rank.

No. ToneBench measures one skill: writing long-form scripts in one specific human voice under a strict style guide. The task depends on careful instruction following and style control, but it is deliberately narrow, and a model can lead this board while ranking lower on code, math or agentic tasks. Use it as one signal alongside benchmarks for the skills you need.

Send missing models, incorrect numbers or disagreements with the methodology to [email protected]. The same address handles requests for a similar evaluation of another brand voice.