The number the marketing leaves out
There is a measurement that let the same models loose on synthetic bugs and on real ones. The difference is not a nuance, it is an order of magnitude.
| Model | Synthetic bug | Real repo, code only | Real repo, with bug report |
|---|---|---|---|
| A | 45.97% | 2.95% | 12.96% |
| B | 37.71% | 3.16% | 9.58% |
| C | 52.90% | 4.59% | 17.81% |
| D | 46.15% | 1.29% | 9.61% |
(Share of faulty lines pinpointed on the first attempt.)
One to five percent. That is what the model is worth if you hand it the code and ask where the bug is.
And the text of the bug report on its own multiplies that three- to sevenfold — by six to thirteen percentage points. The model did not get smarter: you gave it something to hold on to.
The ordering that is worth thirty-seven points
This is the least known and the most practical result in the whole area.
A 2026 conference paper took the same bug, the same model and the same code — and changed nothing but the order things were pasted in:
| Task set | Suspect code first | Suspect code last |
|---|---|---|
| One | 57% | 20% |
| The other | 38% | ~3% |
Same bug, same code. The hit rate falls to a third.
Dumping in the whole file is buying a lottery ticket. Put the suspect part at the start of the prompt.
What helps — and what hurts
This is the summary the measurements add up to:
| Intervention | Measured effect |
|---|---|
| The text of the bug report or fault description | +6…+13 points |
| Pointing at a file or a line | between the solved and the unsolved cases this indicator shows a 27.8-fold difference |
| Runtime evidence — variable values at the moment of failure | +51% in the number of correct fixes |
| The suspect code at the start of the prompt | up to +37 points |
| Raw, long stack trace | can hurt |
The last row is surprising, so it is worth saying precisely: one study names it explicitly as a failure mode that too many suspicious elements coming out of the stack trace scatter the model. What works is not handing over the stack trace, it is handing over the trimmed stack trace.
So "the more context the better" does not hold here. What is relevant helps; what is merely bulky hurts.
And the intervention with the best return is the one fewest people bother with: giving runtime evidence. Not the code, but what the variables held at the moment the bug hit. An improvement above fifty percent — for the price of one print.
Do not tell it what you think
This is the hardest rule of the method to keep, because it is the exact opposite of what instinct tells you to do.
A large-sample measurement — 14,042 questions in five conditions, roughly 70,000 question-answer pairs — looked at what happens when the user mentions an answer. The result:
- Mentioning a wrong answer: up to 15 percentage points of accuracy lost.
- Mentioning a right answer: up to 14.7 points gained.
- Prompted by the user's suggestion, the model revises its answer in 6.2–18.8% of cases.
(Important limit: this was run on multiple-choice questions, not on debugging.)
The measurement closest to debugging is smaller but more telling: twenty-four participants, a real debugging task, two kinds of chatbot — one strongly agreeable, one restrained. With the agreeable version the participants corrected their own false beliefs less often, and spent more time leaning on useless answers.
And the detail that matters most: most of the users never noticed that they were being confirmed in their mistake.
Hence the first rule of the method: the model does not tell you where the bug is — it continues your question. So do not write down where you think the trouble is. Write down what you did, what you expected, what happened — and keep your hypothesis to yourself until the model has offered one.
How general an experience this is: in a developer survey with 33,000 responses the biggest frustration — 66% — was "an AI solution that is almost right, but not quite". And 45.2% say debugging generated code eats more time, not less.
Plausible ≠ correct
Here comes the hardest number, and it lands right at the end of the process.
In automated program repair, researchers checked how many of the fixes that pass the tests are actually correct:
| Tool family | Correct : overfitted | Correct share |
|---|---|---|
| Classic repair tools | 129 : 690 | ~16% |
| Language-model based | 63 : 107 | ~37% |
So 63–84% of the fixes that turn the test suite green are in fact wrong — they simply mould themselves to the tests that already exist.
And worse still: the tools built to spot overfitting do not work either. The same study concludes that "simple random choice outperforms every state-of-the-art detection tool in 71–96% of cases".
There is no machine that decides for you whether the fix is good. You have to read it. The measurements behind that are what the tests as a safety net post goes through in detail.
The method
1. REPRODUCE IT FIRST, WITHOUT AI.
If you cannot trigger it, the model will not either. The
1-5% hit rate measures exactly what happens without this.
2. WRITE THE BUG REPORT, NOT YOUR GUESS. (+6-13 points)
What you did -> what you expected -> what happened.
3. NARROW IT TO A FILE OR A LINE, if you can.
This is the indicator with the biggest gap between the
cases that got solved and the ones that did not.
4. GIVE RUNTIME EVIDENCE.
Variable values at the moment of failure. (+51%)
One print is worth more than a hundred lines of code.
5. SUSPECT CODE AT THE TOP OF THE PROMPT. (up to +37 points)
6. TRIM THE STACK TRACE.
A raw, long trace scatters the model.
7. KEEP YOUR HYPOTHESIS to yourself until the model has
offered one. A wrong guess costs up to 15 points - and
you will not notice.
8. DO NOT ACCEPT "THE TESTS ARE GREEN".
63-84% of plausible fixes are wrong.
Common misconceptions
- "I hand over the code and it tells me where the bug is." In a real repository that is a 1–5% hit rate. Without the text of the bug report it is worth almost nothing.
- "The more context I give, the better." A raw, long stack trace measurably hurts. The relevant part helps, the bulk does not.
- "I write down what I think so it does not set off in the wrong direction." Exactly backwards: mentioning a wrong hypothesis costs up to fifteen points — and most users never notice that they were confirmed in their error.
- "The order I paste things in makes no difference." Same bug, same code, different order: 57% or 20%.
- "The tests are green, the fix is done." Most fixes that pass the tests are overfitted. And among the detection tools, random choice does better.
What to take away
The model was never handed a debugger, it was handed text completion. So in debugging the question is not which model you use but what you give it: a reproduction, a fault description, a location, runtime values — and all of that in the right order. Leave your own guess out. And at the end the same rule holds as everywhere else: a green test is not proof.
Reviewing the generated fix is what the reviewing AI-written code post walks through; scoping the task itself is covered by the first task with a coding agent post, start to finish.