The two numbers that cannot both be right
In the summer of 2025 a report from a research group linked to MIT went round the trade press: "95% of organisations are getting zero return" from generative AI. The report itself notes that its data is "directional, based on individual interviews, not from official corporate accounts" — the sample was 52 interviews and 153 executive questionnaires, the latter collected at four industry conferences. A university researcher in Pennsylvania wrote publicly about it that he had read the document through several times and could not see where the 95% comes from.
Three months later another university research group surveyed more than 800 corporate decision-makers at companies with over 50 million dollars in revenue. The result: three out of four executives reported a positive return, and 72% of them said their company already tracks generative AI metrics tied to profit, throughput or productivity.
The same technology, the same six months, the same method — executives reporting on themselves — and the two results are each other's opposite. If two serious surveys give opposite answers to one question, the problem is not in the technology. It is in the measurement.
The number that gets quoted wrong
It recently spread like wildfire that "94% of companies do not measure AI results". No such survey exists. The 94% comes from a trade-magazine headline that subtracted a consultancy survey's 6% figure: according to that survey, 6% of respondents qualify as high performers — so 94% are not. That is about the absence of return, not the absence of measurement. The two are not the same thing.
On measurement, though, there is a real figure, from the same survey (1,719 respondents across 97 countries, May–June 2026): the statement "we have a defined process for quantifying the impact of our AI initiatives" is claimed by 52% of the high performers and 33% of everybody else.
So roughly two thirds of ordinary companies have no process at all for measuring it. That number is defensible, and it says precisely what is worth saying.
The 40 percentage points: why you cannot simply ask
This is the article's most important piece of evidence, because the same team measured the reality and the perception, on the same people.
In a 2025 randomised experiment, 16 experienced open-source developers worked on 246 real tasks (the tasks were randomly assigned to AI-allowed and AI-forbidden groups), jobs of about two hours on average, in codebases they had worked on for five years on average:
| What was asked / measured | Result |
|---|---|
| Expectation beforehand | AI speeds things up by 24% |
| After the experiment, their own estimate | AI sped things up by 20% |
| Time actually measured | AI slowed things down by 19% |
So the participants felt a 20% speed-up while becoming 19% slower. In the spring of 2026 the same team spelled out the conclusion in its own self-report survey of 349 technical workers: in the 2025 study people overestimated by an average of 40 percentage points the effect of AI on the time they spent on a task. And they added that productivity estimates from surveys are routinely larger than those from field experiments.
There is an honest postscript to this too. In February 2026 the same research group announced that it has to change its experimental design, because 30–50% of the developers said they simply do not submit certain tasks, since they do not want to do them without AI. The control group, in other words, has started to empty out. They think it likely that the speed-up in 2026 is bigger than what they measured in 2025 — but they also state outright that their evidence for this is very weak.
The lesson for your company: if you ask the team how much AI helps, you get back a number with a 40-percentage-point bias. Not because they are fibbing.
What to measure instead of the feeling
Two numbers sitting side by side in one of the big consultancy surveys tell the whole story: 80% of respondents feel their individual productivity has improved, while 37% see any effect on results and 6% a significant one.
The reason for the gap is not mysterious. An hour of work saved is only a saving if somebody genuinely spends that hour on something else, or if it means you do not have to hire somebody. If the day merely got less tight, that will not show up in the results — and it never will.
Five rules of measurement follow from this:
1. A baseline before the rollout, for two weeks. Three numbers, all three from the system: the count, the turnaround time from timestamps, the error rate. If there is no timestamp, the project starts with timestamping.
2. One process, not five. A series of randomised experiments on a global online marketplace tested seven AI applications inside the same company: the pre-sales chatbot lifted sales by 16.3%, search refinement by 2.9%, product-description generation by 2.1% — while the push message and the ad-headline generator lifted nothing. If you roll these out at once, you blur the winners and the losers together.
3. A control, even in a small company. Two people, two similar sets of customers, one with AI and one without, swapping after two weeks. If that is impossible, alternating weeks. If that is impossible too, then all you have is before-and-after — and in that case write into the report that this is a signal, not proof.
4. The metric is the output, not the time. Quotes sent per week, quotes won as a share of quotes sent, time to first reply, the share of work sent back, hours billed with the same number of people. The question "how much time did I save" is the 40-percentage-point trap.
5. The cost side in full. Not just the subscription:
Monthly net gain = (hours of work saved × hourly rate)
− subscription
− (review hours × hourly rate)
One-off rollout cost = rollout hours × hourly rate + training hours × hourly rate
Payback period (months) = one-off rollout cost / monthly net gain
Review time is the item almost everyone leaves out: if somebody spends three minutes checking every output and there are 400 outputs a month, that is 20 hours.
The statistical limit it is better to know in advance
That 246-task experiment had, by its own description, "just barely enough" statistical power to rule out a zero effect. Sixteen people, a professionally run study, 246 observations — to detect one 19% effect.
[Own calculation] If your process produces ten items a week, 246 observations take close to half a year. If the expected effect is half that size (10%), the required sample grows roughly fourfold, because sample size is inversely proportional to the square of the effect size — and that is close to two years.
What follows is that a small company cannot reach statistical significance on its own process, and does not need to try. What you can aim at is the big, visible difference: double the count, half the turnaround. If there is only a small deviation, the honest answer is that it is not measurable — not that "it is noticeably better anyway".
What the rigorous measurements say, when measuring does work
It is worth knowing where an effect gets measured and where it does not, because the difference is instructive.
| Study | Sample | What it measured | Result |
|---|---|---|---|
| Customer-support rollout | 5,179 people | issues resolved per hour | +14% on average, +34% for beginners, ~0% for the experienced |
| Product-development field experiment | 776 people | quality of the solution | an individual working with AI is as good as a two-person team without it |
| Advertising production | 2,234 people, 11,024 ads | output and performance | +50% ads per head, but a more homogeneous output |
| Danish payroll data | national, administrative data | pay and hours worked | any effect larger than 2% ruled out two years after the rollout |
The first three measured the output directly: units, quality, performance. The fourth looked for the effect in pay and hours, and did not find it — while the same study did show that tasks were rearranged and many people changed occupation.
The difference is not in the AI, it is in what you measure.
The common misconceptions
"The 95% means that 95% of pilots fail." It does not mean that. The report claims something about return, on the basis of questionnaires collected at conferences, and it flags itself as directional.
"If it does not show in the numbers, it is not working." Your sample may simply be small. See above.
"If it does show in the numbers, it is working." Only if there was a control. Seasonality, one good month, one new customer — each of them produces a 20% improvement on its own.
"We will measure it at the end." Without a baseline there is nothing to measure. This is the one step that cannot be made up afterwards.
What to take away
On the question of return, surveys are not reliable, because they rest on self-reporting, and self-reporting is off by 40 percentage points. In your own company, though, you can measure — if the measuring is part of the rollout and not a question asked afterwards: two weeks of baseline, one process, some kind of control, an output metric, and review time on the cost side.
And if nothing shows up anyway, there can be two reasons: either there really is no effect, or the process stayed the same and only one step of it got faster. The second one — why swapping the tool is not enough — is the subject of redesigning the workflow. And the build-up of the first three months, baseline included, is laid out step by step by the small-company AI strategy.