Methodology
How AllBenched turns published benchmark results into one ranking, and where that ranking is uncertain.
1. Sources
AllBenched prefers the original publisher of a benchmark, or an evaluator that runs models itself, over leaderboards that copy other leaderboards. Every result keeps its source, its retrieval date and a link back to where it was published. Sources are only used when their terms allow reuse; the full list of sources reviewed, including those rejected and why, is on the sources page.
- Epoch AI Benchmarking Hub (CC BY 4.0): Epoch's own runs, plus results it collects from other benchmark publishers and credits to them.
- LMArena (CC BY 4.0): blind human preference ratings for text, web development and vision, using the style-controlled boards that adjust for answer length and formatting.
- models.dev (MIT): the model registry, release dates, context windows, open-weights status and list prices. It is not used for scores.
Scores that model makers publish about their own models are not used, because nobody independent has checked them.
2. Model identity
Publishers name the same model in many ways: "claude-opus-4-5-20251101", "Claude Opus 4.5 (High, 32K)" and "anthropic/claude-opus-4.5" are one model. Names are normalized (case, separators, provider prefixes, date stamps) and matched through an alias table. Effort and thinking settings such as High, Max or Non-reasoning are split off and kept as separate configurations of the base model, because they often score very differently. Names that cannot be matched with confidence are held in a review queue instead of being guessed.
The leaderboard has two views. Configurations ranks each setting on its own. Model families merges a model's configurations: for each benchmark it takes the best published result of any configuration, since that is what a user of the model can get by choosing the setting. The family view reuses the benchmark difficulty and noise estimated from the configuration view, so both views share one scale.
3. From results to category scores
Raw results are not comparable: 60% on a hard benchmark can mean more than 90% on an easy one. Each result is first moved onto an open-ended scale (log-odds for percentages, the natural Elo scale for arena ratings, the logarithm for money earned). Then, within each category, AllBenched fits a latent ability model similar to item response theory:
result = discrimination × ability + offset + noise
Each benchmark gets its own discrimination, offset and noise level, and each model gets one ability per category. The fit alternates between the two until it converges. A model's ability is a precision-weighted average of the abilities its results imply, so precise and discriminating benchmarks count more, and a model does not need to have taken every benchmark. Benchmarks with fewer than 5 models are left out of the fit. Once a model has 3 or more benchmarks in a category, no single benchmark can make up more than 35% of its score there, so one test cannot decide a ranking.
Three rules prevent double counting:
- One result per model, benchmark and source: when several runs exist (different agents or settings), the best one is used.
- Versions of the same benchmark (for example SWE-bench Verified from two evaluators, or two versions of FrontierMath) share one lineage, and their weight is split between them.
- Copies of results published elsewhere are tracked but never scored twice.
4. One common index
Every score on the site uses the same index: 50 is the average of established model families (those with results in at least 4 categories) and 10 points is one standard deviation among them. This holds per category, so a 65 in Coding and a 65 in Math are equally far above average.
| Category | Reference models | What it measures |
|---|---|---|
| Coding | 199 | Real code: fixing, writing and refactoring. |
| Agents | 185 | Multi-step tool use: terminals, browsers, computers. |
| Reasoning | 225 | Hard novel problems and expert science. |
| Math | 196 | Competition and research math. |
| Knowledge | 166 | Factual accuracy and expert knowledge. |
| Frontend | 132 | Web interfaces people prefer in blind tests. |
| General | 191 | Blind human preference on everyday chat. |
| Multimodal | 126 | Understanding images, documents and charts. |
5. Overall scores and profiles
The overall score is a weighted average of the category scores a model has. Different readers care about different things, so several weightings are offered:
| Profile | Coding | Agents | Reasoning | Math | Knowledge | Frontend | General | Multimodal |
|---|---|---|---|---|---|---|---|---|
| Balanced | 25% | 20% | 15% | 10% | 10% | 5% | 10% | 5% |
| Coding | 55% | 20% | 10% | – | – | 15% | – | – |
| Reasoning | – | – | 45% | 35% | 20% | – | – | – |
| Agentic work | 25% | 60% | 15% | – | – | – | – | – |
| Frontend | 30% | – | – | – | – | 60% | – | 10% |
The Balanced profile starts from the proposed split of coding 30, reasoning and math 20, agents 20, knowledge 10, multimodal 10 and other 10. Frontend work is split out of coding (25 + 5), reasoning and math are kept as two categories (15 + 10), and multimodal gets 5 because only three independent multimodal benchmarks are available, two of them narrow; the freed weight goes to general chat preference. The Budget / value profile sorts by the value score below.
6. Missing data and uncertainty
A missing benchmark is never scored as zero, and a model with little evidence is not pulled toward the average. Instead:
- Every category score has a standard error from the fit. The ± values on the site are 90% intervals.
- When a model lacks categories that the profile weights, its overall score uses the categories it has, and its interval widens by the typical gap between a model's categories (currently 3.9 points) times the missing share of the weight.
- A model has early results when it covers less than half of the profile's weight or has fewer than 3 benchmarks. Early models are ranked by score like the rest, with their rank number shown in amber.
- The rank interval (in brackets) is the range of ranks consistent with the 90% intervals. Overlapping rank ranges mean the data cannot separate those models.
- Confidence is High with at least 6 benchmarks and an interval within ±3, Medium with at least 3 benchmarks and ±6, and Low otherwise.
7. Cost and value
Cost is the list price per million tokens from models.dev, using the model maker's own price when it is listed and the median of other providers otherwise. The blended price assumes 75% input and 25% output tokens, a typical chat and coding mix; real workloads differ, and reasoning models can use many more output tokens than this assumes. Cached and batch discounts are not included.
Value is how far a model's Balanced score sits above or below what models at the same price usually score, using a straight-line fit of score against the logarithm of price (currently 6.9 points per tenfold price increase, across 292 priced families). Speed and cost never enter the intelligence scores.
8. Speed
Speed is not shown. The sources that measure tokens per second and latency either forbid reuse or mix different hardware and providers, so a speed column would not be comparable. It will be added when a source with reusable and comparable measurements exists.
9. Freshness
Each source is checked on its own schedule (daily for the current sources). Unchanged downloads are skipped, changed results are kept as history rather than overwritten, results that disappear from a source are retired, and a source that fails or goes quiet is flagged on the sources page. Rankings are recomputed after every change.
10. How this compares to other aggregators
- Epoch's Capabilities Index also fits a latent ability with item response theory; AllBenched does this per category and then combines categories, so profiles are possible.
- Artificial Analysis runs every model on a fixed set of tests and averages them with fixed weights, which needs no imputation but limits coverage to models it can run.
- Some aggregators shrink scores of models with few results toward the average. That makes a new model with one excellent result look mediocre. AllBenched keeps the estimate, widens the interval and marks the model early instead.
11. Known limitations
- Coverage follows what publishers test: new or niche models often have few results and stay early.
- Several benchmarks are close to saturation, which limits how well they separate the strongest models; the fit gives them less weight but they still count.
- Human preference arenas measure what voters like, which overlaps with but is not the same as correctness.
- Results from different agents or harnesses are treated as the same benchmark, keeping the best published run.
- The intervals cover measurement noise on the benchmarks that exist. They do not cover the choice of benchmarks or of profile weights, which is why the profiles are shown side by side.
The scoring code lives in the project repository (not public yet): https://for.dd0.org/vlad/allbenched.