The independent measurement
There is a March 2026 measurement that deliberately goes after real defects rather than style: 174 verified pull requests, with signal-to-noise ratio among its metrics.
| Setup | Recall | Precision | Signal-to-noise |
|---|---|---|---|
| Single-pass, strong model | 27.01% | 3.56% | 5.11 |
| Single-pass, small model | 18.39% | 3.51% | 2.89 |
| Self-reflecting, strong model | 32.76% | 5.10% | 1.95 |
| Self-reflecting, small model | 27.59% | 3.19% | 0.91 |
Read two things out of it.
The first: it finds 18–33% of the real defects, and 3–5% of what it says is real. One in twenty comments.
The second, and the one that matters more: the self-reflecting agent improved recall — it found more real defects — but, in the authors' words, it "pays a steep price in signal integrity at the same time". In the worst case the signal-to-noise ratio is 0.91: more noise than signal.
This is the closest thing we have to a measurement of review fatigue. What wears a reviewer down is not the number of defects, it is the number of false alarms.
(An important limit: this measurement looks at general model and agent paradigms, not at the named commercial products.)
The vendor measurement — peer-reviewed, and saying something else
There is another number that deserves to be taken just as seriously: a large developer-tooling vendor published a one-year evaluation on its own platform, and the study appeared in a peer-reviewed conference proceedings.
- 38.7% of the generated comments triggered a code change in the following commits.
- PR turnaround got 30.8% shorter.
- 35.6% fewer human-written comments were needed.
On the face of it this is the exact opposite of the previous section. It is not.
Why the two numbers do not contradict each other
There is a measurement for this too, and it is the most important methodological part of this post.
A 2026 study looked at 2,604 bot-generated comments that real engineers had labelled "fixed" or "won't fix". Then it checked how closely an automated, model-run evaluation agreed with those labels: agreement between 0.44 and 0.62.
The explanation, verbatim: developer decisions "reflect not only comment quality but also contextual constraints". According to the interviews, labelling behaviour is "strongly influenced by workflow pressure and organisational constraints".
In plain terms: whether someone fixed a comment is not the same as whether the comment was right. A busy developer fixes the easy comment and lets the hard one go — regardless of which one mattered more.
So 38.7% and 3.2% are not the same metric. One measures how much the bot moves behaviour; the other measures how often it is right.
What it costs
On list prices, the market looks like this today:
- A free tier exists at most vendors, typically with a capped number of reviews, plus a separate free plan for open-source projects.
- Paid plans run roughly $20–72 per person per month on annual billing; there are usage-based, credit-driven models as well.
- Enterprise tier: self-hosting, single-tenant or on-premise deployment, permission management, audit log — prices are rarely published for these.
- An open-source alternative exists too, run with your own model key.
But price is not where this gets decided.
The question that actually decides it: what the bot sees, and whether it trains on it
This is where the post earns its keep, because vendor defaults are not the same — and they differ in the contract.
One vendor's documentation says, word for word, that it "never uses customer code for training", whatever the data-retention setting is. The same page also states that it shares the code with the model providers — solely for the purpose of the review.
Another vendor at a similar price point writes this: it "may aggregate and anonymise customer data… to train and improve artificial intelligence algorithms and models" — and adds that this can be switched off in the account settings.
Same price level, two opposite defaults. With closed source, that difference decides it, not the feature list.
Worth going through before you adopt one:
1. Does it train on my code? Default YES or NO?
And if yes, where do I turn it off?
2. Which model provider does the code go out to, and
under what terms?
3. How long is it stored? What happens if I cancel?
4. Is there self-hosting / own model key / on-premise
deployment? From which plan?
5. Where is the data, geographically?
6. Is there an independent security certification, and
can the report be requested?
How to test it for yourself — on twenty PRs
This is the only method that answers the question for your codebase. Two hours of work.
1. PICK 20 ALREADY-MERGED PRS from the last six months.
Ones where you KNOW how they ended: which produced
a bug ticket, which needed a follow-up fix.
2. RUN THE BOT ON THEM, blind - the reviewer must not
see its comments in advance.
3. COUNT THREE THINGS:
- how many REAL defects it found out of the ones you
now know were there -> recall
- what percentage of its comments were real
-> precision
- how many comments were pure noise -> this is what
your reviewers will absorb
4. COMPARE that with what your team found anyway.
If the bot only finds what a human finds too, it is
not catching defects, it is taking time.
5. DECIDE ON A NUMBER, not on the demo.
The demo always goes well - on hand-picked PRs.
What has to be admitted
For honesty's sake, three things we do not know:
- There is no independent precision and recall measurement of the named commercial bots on real pull requests. The independent study examines paradigms, not products; vendor comparisons are marketing material.
- There is no independent measurement of whether machine review lowers the share of defects that escape. The vendor study measures turnaround and comments that triggered a code change, not defect rate.
- Review fatigue is not quantified in the literature. The closest indicator is the signal-to-noise ratio above.
Common misconceptions
- "The bot gets 38% of its comments right." That measures how many comments produced a code change — not how many were correct. The precision measurement shows 3–5%.
- "The more defects it finds, the better." The self-reflecting setup found more real defects, but the signal-to-noise ratio fell to 0.91 on the way: more noise than signal.
- "It is safe because I pay for it." Of two similarly priced vendors, one never trains on customer code and the other may do so by default. They differ in the contract.
- "The demo was convincing." The demo runs on hand-picked PRs. Twenty of your own already-merged PRs will tell you the truth in two hours.
- "It replaces human review." The measured benefit is in turnaround and in the number of human comments. The responsibility has not moved.
What to take away
AI in code review does not improve the defect rate, it improves turnaround — even the best measured result is about that. What you get: faster first feedback and fewer small human comments. What you do not get: any guarantee that the real defects have been found, since it catches only a third to a fifth of them.
Two things should decide the adoption: what the contract says about your data, and what twenty of your own already-merged PRs show.
What else is worth going through by hand is laid out in the reviewing generated code post, point by point; for the numbers on the security side, see the security bill of AI code post instead.