How big a task gets through?
There is a measurement for this, and the metric is elegantly simple: the task length — measured in human working time — at which the model reaches a 50% success rate.
According to the measurement from early 2026 the leading pack sits at around 320 minutes, but the confidence interval is wide: between 170 and 729 minutes. The trend is fast: the doubling time has shortened from the earlier seven months to about three months.
Two things are worth adding to that.
One is a methodological limit the researchers spell out themselves: for tasks longer than eight hours, the human baseline time was measured in only 5 cases out of 31, and the rest are estimates. So the top end of the scale is shakier than it looks.
The other is the practical reading. Fifty percent is not a good rate. Give it a five-hour task and it will get it wrong half the time — and reviewing five hours of botched work takes you five hours. Your first task should be well under an hour of human work, somewhere the mistake shows up immediately.
What to give it, and what not to
One large provider's official documentation spells out what the agent is for and what it is not. That list is more useful than most professional advice:
GOOD FOR THIS
bug fixing
changing a UI element
increasing test coverage
updating documentation
improving accessibility
paying down technical debt
NOT FOR THIS
broad refactoring that requires knowledge of several repos
a complex task that needs dependencies and legacy code understood
a critical bug AFFECTING PRODUCTION
security, personal data, authentication
a vague or open-ended task
WHAT YOU WANT TO LEARN FROM <- the docs single this one out
The last point is the most important, and the least technical. If you would learn something about the codebase from solving a task, then what you handed over was the learning — not the work.
What to write into the task
The same documentation asks three things of a good task description:
- A clear description of the problem or the work to be done.
- Complete acceptance criteria — what counts as a good solution.
- Guidance on which files need to be changed.
The third is the one most people leave out, even though it is what shortens the exploration most.
The size of the difference is best shown by a before-and-after comparison:
| How most people ask | How it works |
|---|---|
| "write tests for foo.py" | "write a test for foo.py that covers the case where the user is logged out; do not use mocks" |
| "fix the login bug" | "users report that login falls over after the session expires; look at the auth flow under src/auth/, especially the token refresh; first write a failing test that reproduces it, then fix it" |
| "the build is broken" | "the build stops with this error: [error message]. Fix it, and check that it runs. Address the root cause, do not suppress the error" |
The most common mistake: it has nothing to run
This is the point that decides the whole session, and the vendor documentation says it with unusual honesty:
"The model stops when the work looks done. If there is no check it can run, then 'looks done' is the only available signal — and you become the verification loop."
The same thing stated positively: give it something it can run itself — a test, a build, a screenshot it can compare against. That is the difference between watching the session all the way through and being able to walk away from it.
Which is why the order for a first task is: the failing test first, the fix afterwards. Not only because that makes it checkable — but because it also makes the definition of the task itself unambiguous.
The four phases
The working method that has proven itself is four steps: exploration → plan → implementation → commit. The first two run in a mode where the tool reads and runs read-only commands, but does not rewrite your sources.
And a rarely quoted addition from the same documentation: do not always plan. If the scope is clear and the fix is small — a typo, a log line, a rename — just ask it to do it straight away. The rule in one sentence: if you can describe the diff in one sentence, skip the planning.
Five named failure patterns
The vendor documentation names these separately, because they are so common:
- The everything-in-one session. Several unrelated tasks in the same conversation. The context has to be cleared between them.
- The round-and-round fixing. After two failed fixes, do not go for a third: clear the context and write a better task description.
- The bloated project instructions. If it is too long, half of it gets lost. The project instructions post puts numbers on that.
- The trust-verification gap. "If you cannot verify it, do not ship it."
- The endless exploration. Narrow the scope of the investigation, or give it a separate subagent for that.
Behind every one of them is the same thing: the context window fills up fast, and as it fills, performance degrades. Most good practice follows from that single constraint.
What reality shows: 11,048 agent PRs
There is an analysis that looked at 11,048 closed pull requests opened by agents in real repos with at least five hundred stars. Of those, 9,799 were also reviewed by a human.
The merge rate is 63.1%. That is far better than a sceptic would expect — but the details are more instructive.
Among the rejected PRs, the agent itself was the cause of the problem in only 35.7% of cases. The rest were process reasons: duplication, a stale branch, or simply the maintainer not having asked for this. So two thirds of the rejections were not about the code being bad.
And two numbers that give one pause:
- 79.1% of merged agent PRs had no feedback round at all — nobody asked for a change, it just went in.
- According to another analysis, of 40,214 PRs, 77.51% of merged agent PRs were merged by the submitter themselves, against 57.63% of human PRs.
From which follows the most practical piece of advice in this post: do not merge your own agent PR on your first task. Have someone else look at it. By the numbers, this is the step that gets skipped most often.
That same analysis also throws up an odd paradox: one extra reviewer comment raises the merge chance of a human PR by 2.7%, but lowers an agentic one's by 2.8%. On a human PR, a lot of comments are a dialogue; on an agent PR, they are a warning.
The benchmark nobody should base a purchase on
The most frequently quoted coding benchmark is built from real GitHub issues: you get a repo in a given state plus a bug description, and you have to write a fix that passes the tests of the original fix.
What it does not measure: design, architecture, the interface, performance, maintainability — and everything else that cannot be checked with an existing test.
There are two things worth knowing about it.
It is saturated. In the autumn of 2026 the leading pack sits at around 97% on a 500-item set. At that level the difference across the remaining few tasks can no longer be told apart from noise.
And it had a serious problem. A 2024 analysis of the original, unfiltered version found that in 32.67% of successful fixes the solution was sitting in the bug report or the comments themselves, and a further 31.08% passed because of weak tests. Corrected, the same system's result fell from 12.47% to 3.97%. The human-filtered subset was meant to remedy exactly this — but it does not fully solve the leakage problem either, because the tasks come from public repos.
Where there is still room: on a 2026 measurement built from terminal tasks, frontier models score below 65%.
Common misconceptions
- "The agent solves five-hour tasks." With a fifty percent chance. Your first task should be well under an hour.
- "The more detailed the task, the better." Precision of scope is what counts, not length. If you can describe the diff in one sentence, do not plan.
- "If it ran, it is done." The tool stops when the work looks done. Without a check, you are the only verification loop.
- "It scores 97% on the benchmark, so it solves almost everything." The benchmark is saturated, and it does not measure design, architecture or maintainability.
- "If it passed the test, I can merge it." Three quarters of merged agent PRs are merged by the submitter. On your first task, that is exactly what not to do.
What to take away
Your first task should be small, boring, and come with a check the agent can run itself. Write down what the problem is, what counts as a good solution, and which files are involved. Do not hand over the task you wanted to learn from, and do not merge your own agent PR.
The approval modes and where all this should run are laid out in the choosing a coding setup post, architecture by architecture; the concrete failure patterns of finished code are gathered in the reviewing generated code post, with examples.