Why is it not settled by the announcements?
Because an announcement is the vendor's own measurement, a ranking measures in English on a generic task set, and your colleagues' opinions consist of three memorable cases. None of those is your work.
Choosing a model is really a measurement task, not a research one. It needs neither a data scientist nor a tool — it needs twenty real tasks and someone to count the errors. This article sets that out step by step.
Week one: the task set
This is three quarters of the work, and it is the part people skip. Four rules:
- Twenty tasks, with real material. Not invented examples: copied, actually occurring cases, anonymised.
- The real distribution. If half the incoming cases are easy, then ten of the twenty should be easy. Measuring only the hard cases is exactly as misleading as measuring only the easy ones.
- Include five "ugly" cases. Missing data, contradictory information, bad spelling, two languages mixed in one text. This is where models separate.
- A week of collecting. Do not assemble it in one sitting — over a week, as they come in. That is what makes the distribution real.
How do you write down the "good answer"?
This step decides whether your measurement produces a number or an opinion. For each task you have to write down in advance what counts as a good answer — before you have seen any model's output.
You do not need a word-for-word solution. Three or four points are enough: what it must contain, what it must not contain, and in what form. For example: "Must include the deadline and the owner. Must not invent an amount. At most three sentences."
If for a task you cannot write down what the good answer is, then that task is not measurable — take it out of the set. That is not a failure: it is a useful realisation that what you have there is not a model problem but an undefined expectation. This is the same thought that opens Can AI Do This Task?, a separate article.
Week two: the run
From here it is quick. Four steps, half a day in total:
- Pick three or four models. Not the four most expensive: a cheap tier, a middle tier, a top tier, and if it makes sense, an open-weight one. Through a model router that is one account.
- The same prompt on all of them. If you tune per model, you are not comparing models, you are comparing your own patience.
- Run it twice. The same question twice does not always give the same thing. If the two runs differ, that is a result in itself.
- Save the outputs. Not only the score: the raw answers too. Those become your baseline at the next model change.
The scoring sheet
Five criteria, and they are not equally important. The weight comes from your own task:
| Criterion | How you measure it | When it matters most |
|---|---|---|
| Accuracy | how many of the 20 are right | always |
| Format adherence | how often it holds the requested structure | when a machine consumes it |
| Speed | average response time | when a user is waiting |
| Price | token cost of the 20 tasks | when it runs often |
| Language quality | grammar, terms, register | in non-English work |
One important detail: count format adherence separately, not folded into accuracy. A model can be better on substance and still be unusable in an automated flow, because it does not hold the requested structure every time.
Five mistakes that ruin the measurement
You will meet all of them if you are not watching:
- Writing the good answer afterwards. You will unconsciously shape it to what you saw.
- Using a different prompt per model. The most common one, and it destroys comparability.
- Measuring only the hard cases. Spectacular, but it does not describe your daily work.
- Scoring by impression. "This one is nicer" — people feel the longer, more confident answer is better even when it is worse.
- Measuring only in English. If your work is in another language, that language's performance is the point — and it behaves differently from English.
The decision
At the end you will have a table. There are three typical outcomes, and each calls for a different move.
A big difference in accuracy. Then it is simple: you need the better model, even if it costs more. This is the rarer case.
A small difference, a big price gap. This is the most common. Here the cheaper one wins, and for the few remaining failures you build a cascade: the uncertain cases get passed up to the expensive one.
All of them are weak. That is a result too, and more valuable than you would think. Here the answer is not to switch model but to break the task apart or rewrite the prompt — the five elements of a prompt are a good starting point.
What remains afterwards?
More important than the decision is what you built along the way: twenty tasks with written good answers and saved outputs. That set stays with you for years.
You run it when a new model arrives, when a price changes, when the one you use gets retired, or when someone says "this is the better one now". Half an hour, and you know the answer. The set and the prompts are worth storing the way a prompt library entry is stored, with an owner and a date.
What to take away
A week of collecting, half a day of running, and at the end a number rather than an opinion. The work is not the running but the twenty real tasks and the good answers written in advance — without those the measurement is only an impression. And what remains is worth more than the decision: a test set that settles every future model question in half an hour.