Why is this harder than you expect?
With traditional software a test is binary: it passes or it does not, the output matches or it does not. With an agent three things change at once: the model is not deterministic, the environment changes too (a website, a database, a calendar), and the correct answer is often not a single piece of text.
There is a fourth one on top of that which many people only notice later: the logging standard itself is not finished. The generative-AI part of the industry standard for tracing is still marked "under development" in September 2026 — not stable, not even a release candidate. It has also moved into a separate repository in the meantime, and its reference schema is still marked "to do" in the documentation.
This is not a detail. It means there is no established shape for logging agent runs that you could build on for the long run — vendors work with their own field names, and those are still changing.
The three ways of scoring
The primary documentation distinguishes three modes and ranks them as well. It is worth taking literally, because the order is not what most people would guess:
| Mode | Assessment |
|---|---|
| Code-based | "The fastest and the most reliable, and it scales extremely well" — in exchange, it cannot make fine distinctions. |
| Human | "The most flexible and the highest quality, but slow and expensive. Avoid it where you can." |
| Model-based | "Fast and flexible, scalable, suitable even for complex judgement. Verify its reliability first, then scale it." |
The message of that order: start with code. In most agent tasks there is far more that can be checked in code than you would think — did the file get created? Is the JSON valid? Is the required field in there? Did it call a tool it was not supposed to? These are fast, free and they do not make mistakes.
Model-based judging only comes in where code cannot decide. And — as the next chapter shows — even there you must not trust it blindly.
The model as judge: measured, in both directions
This is the most important chapter in the article, because this is where most of the silent failures are.
The good news. According to a 2023 academic study, strong judge models reach over 80% agreement with human preferences — as much as there is between two humans. A judging pipeline built for a multi-agent failure typology reached 94% accuracy and an agreement coefficient of 0.77 against human annotation, and it generalised to systems it had not seen before as well (0.79). For comparison: in the same place the agreement between human annotators was 0.88.
The bad news, and this is the more important one.
There is a leaderboard measuring browser agents that publishes two separate result columns for the same systems: one verified by a human and one produced by an automated judge.
[Own calculation] I paired the two up, across 13 systems:
| Value | |
|---|---|
| Number of systems | 13 |
| Average difference | −5.49 percentage points |
| Range | from −0.4 to −14.0 points |
| Cases where the automated judge scored lower | 13 out of 13 |
The automated judge does not err randomly but systematically, in one direction: it undervalues. And the difference is not even — even with the best systems it reached 14 points.
(A methodological caveat: the two data sets do not necessarily come from the same runs, so part of the difference is measurement noise. The consistency of the direction — 13 out of 13 — cannot be explained by noise, though.)
The practical consequence: if you judge only with a model, you do not know which way and by how much you are wrong. So it is worth putting human checking alongside it from time to time — not on every run, but on a sample — and looking at how big the offset is. If you know the size of the offset, model judging is usable; if you do not, your numbers are not comparable with anything.
Three rules that make the judge more reliable
The vendor guide is specific:
1. A DETAILED, UNAMBIGUOUS RUBRIC
Not "is the answer good" but: "the answer must
contain X in its first sentence; if it does not,
the answer is automatically wrong".
2. ASK FOR A NUMBER OR A LABEL, NOT AN OPINION
"correct" / "incorrect", or a 1-5 scale. Purely
textual assessment cannot be processed quickly
and in bulk.
3. LET IT THINK - THEN THROW THAT AWAY
Ask it to reason first and only then give a
score; afterwards you discard the reasoning. This
measurably improves the quality of the judging.
What should you measure? — accuracy on its own is not enough
A public leaderboard that takes itself seriously publishes this for every entry, and it is a good template for your own measurement too:
| What to measure | Why |
|---|---|
| Accuracy | the basic question: did it work |
| Confidence interval | without it you do not know whether the difference is real |
| Number of runs | this decides how small a difference you can detect |
| Success rate over several attempts | it measures something different from the single-attempt one |
| Tokens in and out, split out by cache | this is where you see where the money goes |
| Dollars per run | the only number the business cares about |
| Runtime | the limit on the user experience |
Among the success criteria the documentation names eight categories, and most measurements look only at the first: task fidelity, consistency, relevance and coherence, tone and style, preservation of privacy, use of context, latency, price.
"Preservation of privacy" as an evaluation criterion is the item left out most often. And yet it is easy to automate: you search the output for patterns that should not be in there.
How big a sample do you need to detect a regression?
There is a concrete, measured example for this. One leaderboard publishes 330 runs for every entry, and its confidence interval therefore sits between ±2.4 and ±3.9 points.
And the most important practical rule follows from that. If it takes three hundred and thirty runs to bring the uncertainty down to around three points, then at ten runs the uncertainty is above twenty percentage points — uncertainty falls with the square root of the number of runs. So if your agent solves eight out of ten test cases and seven after an upgrade, that is pure noise.
A concrete illustration from the same place: the same model at two thinking levels scores 57.88% and 58.18% — the difference is inside the margin of error, while the price difference is 43%. Whoever paid the 43% for those 0.3 points bought themselves some noise.
Practical recommendation: use fewer test cases, but run each of them several times. Twenty test cases with five repetitions is worth more than a hundred test cases run once.
A model upgrade is the most common silent regression
Here is the most practical part of the article, because it affects everyone who runs a working agent.
In 2026 it is not even optional: the big providers retired the previous model generations one by one. Anyone who set up an agent in 2025 and did not touch it had to switch models in 2026. Handling deprecation is what the when your model is deprecated post walks through, step by step.
And the switch is not neutral. The vendor's migration guide lists, item by item, the behaviour changes that can break a working agent:
| Change | What it breaks |
|---|---|
| Longer default answers | length control goes out of the window; lowering the effort shortens the thinking, but not reliably the visible answer |
| Literal instruction following | the new model does not silently generalise from one item to another, and does not infer a request you did not spell out |
| Over-checking | the new model checks its own work on its own — so a self-check instruction inherited from the old prompt duplicates the work |
| More delegation | it starts subagents more readily, which means more tokens and different failure modes |
| Repriced effort levels | the same setting means a different amount of tokens than it used to |
The third row is the most insidious. If your prompt still contains a "check your work before you answer" sentence that you put in two years ago, then on the new model it does not improve things, it makes them worse — the agent checks twice, and gets slower and more expensive.
There is a technical trap on top of this: on newer models the parameters that govern randomness (temperature, sampling thresholds) have become obsolete, and if you set them to anything other than the default value the call comes back with an error. A client written in 2024 simply falls over on that.
The migration checklist
WHEN YOU SWITCH MODELS, IN THIS ORDER
1. RE-MEASURE COST AND LATENCY
On your own workload, not on the vendor's sheet.
Thinking tokens are billed as output tokens:
what used not to think at all may now
produce a lot more output.
2. TAKE OUT THE OLD LENGTH-LIMITING SENTENCES
Measure the raw length first, and only then
regulate - not with a sentence from two years ago.
3. TAKE OUT THE OLD SELF-CHECKING SENTENCES
The new models check themselves anyway.
An instruction left in causes over-checking.
4. RUN A FRESH EFFORT-LEVEL SWEEP
Do not carry over a setting tuned to the old one.
5. CAP THE NUMBER OF SUBAGENTS
Or say explicitly when delegation is
justified.
6. RUN THE WHOLE TEST SET, SEVERAL TIMES
One run is not a measurement. See the confidence
interval.
Comparing models against each other in practice is what the model bake-off piece goes into in more depth.
Two measurement traps worth knowing in advance
1. Subagents ruin your segmentation. If you break the statistics down by model, be careful: one session can use several models. If you do not filter for the main requests, the helper and subagent calls book the session's result onto a model that did not even do it. What multi-agent builds mean for measurement is set out at greater length by the multi-agent systems post, with its own table.
2. The cost measurement itself has been known to get it wrong. In one documented case, before a certain version the client over-counted tokens and cost where the answer arrived in several parts — roughly one extra full request per part. And in the vendor's own wording anyway: cost metrics are approximations, and for official accounting you have to look at the billing data.
The lesson from both is the same: your measurement system is software too, and it has bugs of its own. So it is worth looking at the important numbers from two independent sources — telemetry and invoice, automated judge and sampled human check.
The minimum setup
If you can spare a single day for this, build this:
1. TWENTY TEST CASES, WRITTEN DOWN
Real tasks from the past month. For each one:
what the input is, what the right output is,
what counts as acceptable.
2. CODE-BASED CHECKS WHEREVER POSSIBLE
Format, required fields, forbidden tool call,
forbidden pattern in the output. That is the base.
3. A MODEL JUDGE FOR THE REST, WITH A RUBRIC
Ask for a label, not an opinion. Reason first,
then throw the reasoning away.
4. EACH ONE RUNS AT LEAST FIVE TIMES
One run is not a measurement.
5. SAVE SEVEN NUMBERS PER RUN
Success, tokens in, tokens out, dollars, seconds,
number of tool calls, error code.
6. TWENTY RUNS REVIEWED BY A HUMAN EVERY MONTH
This calibrates your judge. Without it you do not
know how big your offset is.
7. BEFORE EVERY MODEL SWITCH: A FULL RUN
That is one day of work, and from then on you know whether something has got worse — you are not guessing.
Common misconceptions
- "We have logging, so we are measuring." The log tells you what happened. Evaluation tells you whether it was any good. In the survey fewer than 40% of respondents measured the second.
- "The model judge will tell me." In a comparison across 13 systems the automated judge scored lower in 13 cases out of 13, by 5.5 points on average. Usable — but only if you know its offset.
- "One run is enough to check a test case." The model is not deterministic. Under five runs you cannot tell noise from a regression.
- "An upgrade can only be better." The new model answers at greater length, takes things more literally, checks itself and delegates more. All four can break a prompt that used to work.
- "Accuracy is what counts." Accuracy without a confidence interval is not a number. And dollars per successful run matters at least as much.
- "Observability is a solved problem." The generative-AI part of the relevant industry standard was still marked "under development" in September 2026.
What to take away
Evaluation is missing from most places not because it is hard, but because there is no pressure forcing it — the agent visibly works, until somebody notices that it does not.
Three things you can build in a day: twenty real test cases, code-based checks on everything that can take them, and seven numbers saved from every run. Those three catch most silent degradations.
And one number worth remembering: under five repetitions it is not a measurement but an impression. Most arguments about whether "the model has got worse" are settled by that difference. How benchmark numbers should be read in general is what the reading benchmarks piece is about, and logging for background agents belongs to the scheduled agents post, in detail.