What is a benchmark?
A standardised set of tests run against every model so they can be compared. Some are built from exam questions, some from fixing real software bugs, and some from problems deliberately made near-impossible.
They are useful: without them we would have nothing but marketing material. But there is one thing they are not for: deciding which model you need.
Four things the score does not tell you
1. The test may have leaked into the training
Models are trained on a large slice of the internet. The questions and answers of popular tests are on the internet. If a model saw them during training, a good result is partly memory rather than capability.
Vendors do try to filter this out, but it cannot be done perfectly. That is why a model can shine on a well-known test and stumble on an equally hard but unfamiliar task.
A useful handle: results on recent, newly built tests are worth more than results on tests that have been circulating for years.
2. One number hides many sub-scores
Most benchmarks measure dozens of sub-areas and average them at the end. Two models with the same overall score can be strong in completely different places: one at mathematics, the other at reading comprehension.
If a detailed breakdown exists, look at that — and within it, only at the slice that concerns you.
3. It does not measure what matters in daily use
Tests typically run in a one question, one answer format. In real work, what matters is this:
- does it follow exactly the format you asked for
- what does it do when the instruction is contradictory
- how consistent is it on the same question
- how does it perform in your language — almost no major benchmark measures this seriously
- how much time and how much money one answer takes
Those last two matter especially: two models with identical scores can run at a fivefold difference in cost.
4. The top of the leaderboard and real usage are two different worlds
If you look at which models are actually used the most, the list is not the same as the top of the leaderboard. In practice, most work is done by mid-sized, cheap, fast models — because they are good enough and cost a tenth as much.
This is not user ignorance. It is the right decision.
What works instead: your own test set
Half a day of work, and it serves you for years afterwards.
1. Collect twenty real tasks. Not invented ones — things that actually came up in your work. Include five easy ones, ten typical ones and five that have never gone well.
2. Write down what makes an answer good. One or two sentences per task. This is the hardest step and the most valuable one: it reveals that in many cases you did not precisely know what you expected either.
3. Run them on every candidate with the same prompt. Unchanged — if you tune the prompt per model, you are not comparing the models.
4. Score blind. Shuffle the answers and score them without seeing which came from where. Without this, you will rate the one you had higher hopes for as better.
5. Note the time and the cost too. The decision is rarely about the best answer, and far more often about what you get for the price.
What emerges at the end is not which model is best, but which is the cheapest one that is still good enough. That is the right question.
One thing worth deciding up front
Run the test on one large and one small model. If the small one delivers the same, work with that — the difference adds up to real money over a year. If it does not, you now know what you are paying the extra for.
Those same twenty tasks are good for something else too: they become your regression test set. When the vendor updates a model or retires one, you run them and know within half an hour whether anything got worse. Without that, you only find out when a customer tells you.
What to take away
Benchmarks are good for seeing where the field is. They are not good for deciding what to use. That decision is made on twenty real tasks, scored blind, cost included — and it takes half a day.