Towards AITowards AIToneBench

Thinking levels · NVIDIA

Does Nemotron 3 Ultra 550B write better when it thinks harder?

Not in any way the board can measure. Nemotron 3 Ultra 550B's default and its high reasoning setting land 1 Elo apart, and their ranges sit almost on top of each other. Cost and speed barely move either ($0.017 against $0.018 a script), so flipping the flag changes nothing you'd notice.

X axis

Both settings we ran

SettingEloStepOverallCost / scriptTime / scriptReasoning tokensBoard rank
high1175 ±66-75.6$0.0180.3 min529#131
default (no effort flag)1176 ±56-75.8$0.0170.4 min539#130

The 95% Elo ranges overlap: Nemotron 3 Ultra 550B runs from 1119 to 1232, and Nemotron 3 Ultra 550B (reasoning) from 1104 to 1236.

Which setting should you use?

I'd leave it on the default. The default run is at #130 and the reasoning run at #131, a gap so small it's noise, so I wouldn't change a setting over it. And if your setup already turns reasoning on for other tasks, there's no reason to switch it off for scripts. It won't help, but it won't hurt either.

Other NVIDIA ladders

Elo comes from comparing every pair of models on every script across the full board, so these settings sit on the same scale as the leaderboard. Cost is one full script at API list prices, and time is one accepted script, retries included. Compare this ladder with other models on the thinking-levels page, or see every NVIDIA model side by side.