The measurement that overturned a piece of received wisdom
A study from February 2026 asked exactly the question everyone treats as obvious: does a repo-level context file help a coding agent?
The methodology is careful. Two task sets: one of their own, with 138 tasks from twelve recent and little-known repos — deliberately unpopular ones, so the model's memory would not help — plus an established 300-task set. Four agents, four models. And two kinds of context file: machine-generated, and hand-written by a developer.
| Context file | Success rate | Cost |
|---|---|---|
| Machine-generated | −0.5% and −2% | +20% and +23% |
| Written by a developer | better than the generated one with all four agents; improves things with one exception | up to +19% |
The step count went up in every configuration, by two to four steps on average. And there is no effect on navigation either: the context file did not meaningfully reduce how many steps the agent needed to reach the relevant files.
The authors' conclusion, word for word: "repository tours — although popular and recommended by model providers — do not help."
And there is one partial result that gives the game away more than anything else. When the documentation was removed from the repo, the machine-generated context files improved performance by 2.7% on average — and came out better than the developer-written documentation as well.
Which means the file is useful when there is no other documentation. Otherwise it just duplicates.
This does not contradict the other measurement
Worth clearing up right away, because at first glance the two results look like they collide.
Another, paired experiment found that a project instruction file cuts token consumption and run time by 20%. That measurement used the developer-written files already present in the repos.
The present measurement, meanwhile, shows that the machine-generated file makes things worse and more expensive.
Together the two give one lesson: what matters is not that the file exists but who wrote it and what is in it. The generated repo tour is the worst case — long, visible from the code anyway, and sitting in the context on every single run. More on that in the project instructions post, with the numbers.
What this costs in money
Let us work it out for a team. Say you run 300 agent tasks a month, at a base cost of half a dollar per task.
Base cost: 300 × 0.50 = $150 / month
With a generated file (+23%): 150 × 1.23 = $184.50 / month
Extra cost: $34.50 / month
Change in success rate: −2 percentage points
Solutions lost: 300 × 0.02 = 6 tasks / month
(Our own calculation from the published ratios.)
Thirty-five dollars extra a month, and six fewer tasks solved. Not a catastrophe — but the opposite sign from the one you did all this for.
Bad documentation is worse than none
This is the post's most important claim, and an experiment backs it.
A 2024 study handed over the same 164 functions in five variants: with no documentation; with the original documentation; with documentation taken from a different function, so it did not fit; with rewritten variable names; and with partial documentation.
Two results came out of it:
- Wrong documentation hurts badly. With random, ill-fitting comments the success rate fell to 22.1% on the weaker model and 68.1% on the stronger one. Statistically significant.
- Missing documentation barely matters. Between the commented and the comment-free baseline there was no significant difference.
To this add another measurement, on a completely different task: there, removing the comments improved the models' hit rate.
Two independent measurements, the same direction. Documentation you are not going to keep up to date is not a neutral liability, it is an active risk. Anyone who thinks documenting is free has the arithmetic wrong.
A stale comment is most dangerous when it is fresh
This is the finding that runs hardest against instinct.
According to a study of 32 Apache projects, changes where the code and the comment drift apart are 1.52 times more likely to lead to a fault-inducing commit within one week, and 1.14 times more likely within two weeks.
And here is the point: the effect is largest immediately after the drift, and decreases over time.
A stale comment is not dangerous because it is old. It is dangerous while someone still believes it. After that the team learns to ignore it — except that the model never learns to.
What to put in, and what to leave out
The vendor recommendation and the measurement above land in the same place. This table is directly usable:
| In | Out |
|---|---|
| Commands it cannot guess | Anything it works out from reading the code |
| Style rules that differ from the default | Standard language conventions |
| Testing instructions, the preferred runner | Detailed API descriptions — link to those |
| Repo etiquette: branch names, PR conventions | Frequently changing information |
| Project-specific architectural decisions | Long explanations, tutorial material |
| Environment quirks, required variables | A file-by-file description of the codebase |
| Non-obvious traps | Self-evident advice |
The right-hand column is essentially what the measurement calls a "repo tour" — the thing it showed to be useless.
The self-test question for every line: "If I deleted this, would it slip up?" If not, delete it.
What does not go stale: the decision log
There is one type of documentation that by definition does not age: the architectural decision log.
The logic is simple. An entry records one decision at one point in time, together with its context and the alternatives that were weighed. If the decision later changes, you do not rewrite it — you write a new entry that supersedes the old one.
That gives you documentation whose upkeep is not an obligation but a new line. And it stays true: the old entry is accurate even when the decision no longer holds, because it records what we thought then, and why.
(An honest limit: we found no measurement of the value of a decision log. What exists is a comparison of templates in a 33-person student experiment — there was a significant difference between the two templates examined, but you cannot draw an industrial conclusion from that sample.)
What can go stale, make runnable
This is the post's most practical mechanism.
An example written into the documentation stays true if it runs. The Python standard library has a tool for this: it picks the interactive examples out of the documentation text, runs them, and compares them with the expected output as written. Other languages have something similar, where the documentation's examples run as part of the test suite.
This is the only form in which stale documentation shows up as a red test instead of rotting quietly.
The same logic extends further: if an explanation is about some behaviour, then that behaviour should have a test — and the test's name should be the explanation. Writing tests is what the safety net of tests post covers in more depth.
The recipe
1. DO NOT GENERATE A REPO TOUR.
Measured effect: -0.5 to -2 points of success rate,
+20-23% cost, +2-4 extra steps per task.
2. WHAT IT WORKS OUT FROM THE CODE, DO NOT WRITE DOWN.
Ask of every line: "if I deleted this, would it slip up?"
3. WHAT CAN GO STALE, MAKE RUNNABLE.
Example in the documentation -> a running test.
Explanation of a behaviour -> a test with a talking name.
4. RECORD THE DECISIONS, NOT THE STATE.
Decision log: context, alternatives, decision, date.
On a change a NEW entry, not a rewrite.
5. DELETE THE STALE COMMENT, DO NOT LEAVE IT THERE.
Wrong documentation is measurably worse than none.
6. FIX IT WHEN THEY DIVERGE.
The tendency to cause faults is at its highest right
after the divergence, not months later.
A number that is missing
In fairness: we looked, and we found no measurement of what share of machine-generated documentation is wrong or out of date.
What exists is a measurement of the effect — that wrong documentation hurts comprehension. On the frequency there is no published data. Anyone quoting a number like that is probably guessing.
Common misconceptions
- "The agent needs documentation to understand the repo." The measurement says the repo tour does not help, and has no effect on navigation either.
- "I will generate an overview, at least then there is one." Two percentage points worse on success rate and 23% higher cost. Nothing would have been better.
- "A stale comment becomes dangerous over time." The other way round: the tendency to cause faults is highest immediately after the drift.
- "Missing documentation is a serious handicap." In the controlled experiment, missing documentation caused no significant degradation. Wrong documentation did.
- "Documenting is free." Bad documentation is an active risk, and a context file charges you for itself on every run.
What to take away
The mark of good documentation in 2026 is not how much it covers but what happens to it when the code changes. What is runnable speaks up: the test goes red. What records a decision does not age. What is a repo tour is better left unwritten, according to the measurement — the model will read the code anyway, only now you are paying for it too.
How an agent finds its way around the code instead of the documentation is what the understanding a legacy codebase post explores, thread by thread.