Browser and Computer-Use Agents: Real Uses

Two very different things under one name

When people talk about a "browser agent", they are really talking about two solutions that have very little in common:

• Pixel-based: the agent gets a screenshot and clicks on coordinates. It works the way a person does — it sees the interface and presses the button.
• Code-based: the agent does not click, it writes automation code (a browser-driving script, HTTP calls) and runs that.

For a long time the first one looked like "proper" computer use and the second like a workaround. A research result from May 2026 turned that around. On the same test of 300 tasks across 136 real websites, the code-based approach reached 86.7% with the same model, while the screenshot-based baseline performed "substantially worse in every difficulty band".

On longer tasks that span several websites the difference is even more striking: the code-based approach reached 60.1%, while the bare baseline of the very same model came in at 33.5%. That is nearly an 80% relative improvement purely from driving the browser differently — the model stayed exactly the same.

The researchers' explanation is easy to grasp: the code-based harness "gives the agent room to choose its own path through the task space". Clicking is linear: one step, one image, one decision. Code can branch, it can loop, and it can settle in fifty lines what would take fifty clicks.

The practical conclusion worth remembering: the absence of an API does not mean you need a pixel agent. The order is: API → code-driven browser control → and only as a last resort screenshot clicking.

What can they do today, measured?

The yardstick for computer use is a test of 369 tasks running on real operating systems. Its timeline is the steepest curve of the past two years:

(Self-reported results on a public leaderboard; no third party verified them. Reading benchmarks has a piece of its own: the how to read an AI benchmark post is worth finishing before you take any of these numbers seriously.)

Two things matter behind this. One is that some of the best results come not from a model but from an agent framework: several attempts, voting between the outputs, specialised control. The other is that these numbers are produced with a maximum step budget.

The step-budget trap

This is the most important table in the article, because it speaks directly to your wallet. Same model, same task set, only the permitted number of steps changes:

Going from 15 to 50 brings +12.7 points. Going from 50 to 100 brings minus 2.5 points. That is not a typo: past a certain point more steps do not help, they only consume — the agent wanders for longer, piles more noise into its context, and finds its way back less easily.

Another measurement points in the same direction: the first fifty steps deliver 82% of the accuracy, and the next fifty add only 3–4 further points — so there the result did not get worse, the improvement simply stopped in practice.

The rule that follows: set a step cap, and not the largest one. Around fifty is where the curve flattens out for most tasks.

On the live web: the numbers, and the caveat

Results measured on real, live websites have improved just as visibly:

But this table has to be read carefully. The 2026 results above 90% were submitted by individual companies, and they were not validated by the same organisation that validated the earlier ones. It is not the same measurement regime — so the rows are only comparable with each other up to a point. The trend is real; the top number is not a promise.

The cost: it grows with the square

Here is the part nobody mentions in the demos.

The computer-use tool set on its own adds roughly 4,500 input tokens to every single request — that is the bare entry fee, before anything has happened. On top of it comes 1,000–1,800 tokens per screenshot. And because the agent resends the whole conversation so far at every step, the context grows step by step.

[Own calculation] At today's price for a frontier model ($5 per million input and $25 per million output tokens), without caching, with one screenshot per step:

Going from 15 to 100 steps is 6.7 times the steps — and 34 times the cost. That is the shape of the curve, not one provider's price list.

Prompt caching takes this down to roughly a tenth and it is worth using — but it does not change the shape of the curve. The quadratic growth is a property of the system: at every step all the earlier screenshots are still sitting in the context.

Two cheap moves that matter a lot: a smaller resolution (1024×768 or 1280×720 is the recommendation for desktop tasks; do not go above 1920×1080), and fewer images — if a step does not need a fresh screenshot, do not ask for one. The logic of pricing in general is what the AI pricing decoded piece covers at length.

And what does one accuracy point cost?

There is a leaderboard that publishes the tokens, the dollars and the time for every run — that is rare, and very instructive. Two findings come out of it, and both are measurable in money:

1. The highest thinking level is not worth it. The same model at "high" and at "maximum" scores 57.9% and 58.2% — the difference is inside the margin of error, while the maximum setting is 43% more expensive and 32% slower.

2. Picking the wrong model is not a few percent, it is an order of magnitude. One model in the field spent $29 per run for a 12.4% result; the best value-for-money setting spent $7 for 57.9%. Per point of accuracy that is a nineteen-fold difference.

The lesson is not which model to pick — that changes monthly — but whether you measure these two numbers at all: dollars per run, and dollars per successful run.

The time nobody budgets for

This is not a defect, it is the nature of the thing. The vendor's own documentation says as much: today's computer-use latency "may be too slow" for human interaction, so it is worth using where speed is not critical — background information gathering, automated software testing, a trusted environment.

That sentence is the best compass there is for this technology. Anyone who wants to run a computer-use agent in front of a live customer has picked the wrong task.

Where does it fall over?

Here is the most surprising figure in the article. The maintainers of the 369-task measurement published the bug reports they had received: 360 reports, on 217 distinct tasks — so more than half the tasks drew a complaint.

[Own calculation] The reports by category:

(Of the 360 reports 153 were left uncategorised, and one report can fall into several categories — which is why the table does not add up to 360. By construction the list skews towards flaws in the measurement: people report what looks broken. The lesson holds regardless.)

The single most instructive case on the list: the agent did the task correctly, but the evaluator looked at a settings file that the program had not yet written out to disk — so the same run counted as a success sometimes and as a failure other times.

And here is the real business lesson. If purpose-built evaluators working on a benchmark got the success verdict wrong 148 times, then in your process too the hardest question is not whether the agent can do it — but whether you can tell that it did. That is exactly why it cannot be run unattended; the human in the loop post walks through where the approval belongs.

The remaining categories describe precisely what makes a production browser agent unstable: the website changes, the login expires, the bot defence kicks in, and a pop-up ruins the click. These are not model errors, and they do not improve when you switch models.

The security part you cannot skip

A browser agent by definition reads untrusted content while it has access to your logged-in accounts. That is the worst combination there is.

Measured numbers from one vendor's own study across 123 test cases and 29 attack scenarios:

Among the browser-specific attacks: invisible form fields hidden in the DOM, and instructions buried in the URL text and in the tab title.

And one documented, real case from the summer of 2025. The malicious instruction sat in a comment on a social site, hidden behind a spoiler marker — white text on a white background, an HTML comment, invisible elements. When the user clicked "summarise this", the assistant read out the email address tied to the account, attempted a login on a deceptively similar domain name, pulled the one-time code out of the mailbox, and wrote it back into the attacker's comment. The fix came in two waves, and the first one was not complete.

The researchers' conclusion is worth quoting: traditional web security assumptions do not hold for agentic AI.

The vendors admit this themselves. One set of documentation says that "under certain circumstances the model will follow instructions found in the content even when they conflict with yours". Another vendor's developer guide states the correct default outright: screen content must be treated as untrusted — text in a web page, a document or a tool response grants no authority.

There is an unsettling side note as well: on the results page of the best-known public, academic prompt injection measurement the most recent entry is from February 2025. Public, independent security measurement, in other words, has fallen behind the models — while the capabilities tripled in eighteen months.

The principles of defence are set out in the least-privilege agent post, and the mechanics of the attack in the prompt injection basics introduction.

The legal part: new, and important

In September 2026 a US decision became final, the first serious precedent on who is liable for what a browser agent does. A large commerce platform sued the developer of an AI browser because its agent logged into the user's account and made purchases on the user's behalf.

The appeals court dismissed the claim, and a few sentences of the reasoning are worth keeping:

"However advanced the assistant may be today, it is a tool, not a person for the purposes of the statute."

"It is the user who accesses the system, with the assistant's help."

The court framed itself deliberately narrowly: it stated that it was not creating a new legal regime for agentic AI, and it did not rule on how other kinds of claim (damages, for instance) would come out.

There are two practical consequences, and both of them matter if you are thinking about this as a business:

1. Terms of service are still very much alive. The court noted in a separate footnote that the decision does not touch the platform's right to govern access through its own contract terms. All it said was that computer-crime statutes cannot be the basis for it.

2. The core of the dispute was not that the agent logged in — but that it hid the fact that it was an agent. The technical question in the case was an identifier string that would have signalled that the user had activated an AI agent, and that the platform could have used to block it. Leaving it out was what the whole dispute centred on.

So if you build a business on a browser agent, the risk is not primarily criminal but contractual: your account can be suspended if the provider's terms forbid automated access. In Europe the AI Act's transparency requirements come on top of that, and their first compliance deadline is 2 December 2026.

When is it worth it?

YES, IF
- there is no API, and there never will be;
- the task can run in the background, nobody waits;
- the step count can be kept under 50;
- a mistake is cheap and reversible;
- you have a separate account you can restrict.

NO, IF
- there is an API - then use that;
- a customer or a colleague waits for the answer;
- money moves, or the step cannot be undone;
- the provider's terms forbid automated access;
- you cannot tell from a run whether it worked.

And an order of preference that follows from the measurements, not from opinion:

1. API, if there is one. Cheaper, faster, safer.
2. If not: CODE that drives the browser.
   Measured: more accurate AND cheaper.
3. Only as a last resort: screenshot and click.
4. In all three: step cap, separate account,
   narrow permissions, logging.

Common misconceptions

• "If there is no API, then you need a pixel agent." No. Code-driven browser control comes first, and the measurement says it is both more accurate and cheaper.
• "I will let it take more steps and it will get there." Above fifty steps the measured result got worse. A longer budget is not persistence, it is more opportunity to get lost.
• "An 85% benchmark means it will do eight out of ten for me." No. That number comes from carefully defined tasks, with a maximum budget and repeated attempts. Your process is nothing like that.
• "Cost is proportional to runtime." Token cost grows with the square of the step count. That is why "just a few more steps" typically means a multiplied bill.
• "It runs on my own machine, so it is safe." The exact opposite: your logged-in accounts are there. The risk of a browser agent does not come from where it runs, but from the fact that it reads untrusted content with permissions in hand.
• "A court ruled that it is allowed." One US decision ruled that certain criminal provisions cannot be applied here. The providers' contract terms were left untouched.

What to take away

Computer use went from 12% to 86% in two years, and in December 2025 it crossed the human baseline. The capability is real — the economics are the bottleneck.

Three numbers worth keeping in your head: step count raises cost quadratically, above fifty steps the result gets worse, and the most expensive setting is not the best one — in one measurement the maximum thinking level cost 43% more for a difference that sat inside the margin of error.

And the most important one: a browser agent is a good choice when there is no API, the task can run in the background, and a mistake is cheap. If any one of the three fails, this is probably not the tool you are looking for. What separates a tool-calling agent from a clicking one is the subject of the tool use and function calling explainer, and the standardised way of wiring tools up is covered by the MCP primer.