Two lists that never match
There is one list of which model is best. And there is another of which models people use most. Anyone who has seen both has noticed that the two do not even resemble each other.
That is not a contradiction but a consequence. A ranking measures what a model can do on a set of tasks. Usage data shows what people pick when they also have to pay for it, when the answer has to be fast, and when it has to work tomorrow too. The two are rarely the same. How to read rankings at all is what How to Read an AI Benchmark covers.
One concrete snapshot
The model routers publish how many tokens run through them on each model. This is a rare occasion: not a marketing number but actual traffic. In September 2026, at one of the largest such services, the top of the list looked like this:
- In first place, a Chinese vendor's preview release, with close to 20 trillion tokens.
- In second, a Western vendor's cheapest tier — not its flagship.
- In third, fourth and fifth, open-weight Chinese models, typically their fast, cheap variants.
Four of the top five places went to open-weight models. Not one Western flagship was among them — even though those are exactly what sit at the top of the rankings.
Four reasons the leader is not what people use
The explanation is not that developers are unaware of the better models. There are four very prosaic reasons:
- Price. Bulk work goes to the cheapest thing that clears the quality bar. At a twenty-five fold price gap that is not stinginess but arithmetic — which is what picking the right tier is about.
- Speed. Where the user is waiting, a fast middling model is a better product than a slow clever one.
- Availability. In a production system what counts is that the call always completes. Anyone building a fallback keeps several models, and the traffic spreads out.
- Switching cost. What works does not get replaced monthly. Around an established model sit prompts, examples and checks that would all have to be retuned.
What the leaderboard genuinely tells you
Do not throw the leaderboard away — just know what it is for. It has three real uses.
It shows you the ceiling. If even the best model reaches only 60 per cent on a task type, your task will not run at 90 either. This is the most valuable piece of information, and it is not available anywhere else.
It shows you the floor. If a model is conspicuously weak on a basic capability, that is worth taking seriously. The negative signal is more reliable than the positive one.
It narrows the field. You cannot try everything out of five hundred models. A ranking is good for getting to five candidates that you then actually measure.
Where the leaderboard misleads
Four specific distortions worth keeping in mind:
- Rewarding formatting. Rankings based on human voting tend to reward the longer, bulleted, confidently phrased answer — which is not the same as the accurate answer.
- Data leaking in. Popular test sets sooner or later end up in the training material. A high score on an old benchmark is therefore worth less than one on a freshly assembled set.
- The vendor's own measurement. The figures in announcements were produced with the vendor's own settings. Not false, but not comparable either.
- Language. Almost every public ranking measures in English. They tell you nothing about performance in a smaller language — that has to be measured separately on your own tasks.
The usage data is not clean either
For the sake of honesty: the traffic list above is not the whole truth either. It shows the data of one intermediary, where developers run bulk tasks — not what an office worker uses in a chat window.
There the ordering would be quite different: on consumer interfaces the big brands dominate, and price sensitivity barely matters, because the subscription is fixed. So it is not one true list and one false one — it is several lists, each answering its own question.
Your own leaderboard
The only yardstick that belongs to your decision is your own. It is not hard to assemble:
- Twenty real tasks from your own work, hard cases included.
- A written good answer for each — this step is the work, and this is the one people skip.
- Three or four models, preferably from different tiers and vendors.
- Four columns at the end: how many good answers, how much time, how much money, how many conspicuous errors.
Those twenty tasks will then stay with you for years: on a model change, a price change or a new release you run the same set and know the answer in half an hour. Run a Two-Week Model Bake-Off sets this out as a detailed recipe.
What to take away
A leaderboard measures capability, usage data measures decisions — and in the decision, price, speed and reliability weigh more than the last few percentage points of accuracy. A leaderboard is good for showing the ceiling and narrowing the field; the answer comes from your own twenty tasks.