Three definitions, three dividing lines
The word "agent" is used today with three different dividing lines, and it is worth knowing which one you are hearing.
| Where it draws the line | Is a chatbot an agent? | |
|---|---|---|
| One of the large model developers | who controls the path of execution — pre-written code or the model | no, if a code path drives it |
| Another one | whether it gets the job done on its own instead of you | explicitly not |
| The university team behind the benchmark | whether there is a loop at all | no, because there is no loop |
The second formulation is the most useful, because it draws the boundary negatively: an application that uses a language model but does not hand it control of the workflow — a plain chatbot, a single-turn query, a sentiment classifier — is not an agent.
In the first one, by contrast, workflow and agent are not opposites but two ends of the same scale. This is not hair-splitting: if somebody is selling you an "agent", it is worth asking which definition they mean by it.
What all three have in common: a loop, tools, and feedback from the environment.
The loop: a hundred lines that tell you everything
There is a public, production agent used by research groups and large technology companies alike, and according to its official description it scores above 74% on a well-known software engineering benchmark. Its entire control logic is about a hundred lines.
Condensed, from the actual source:
run():
messages = [system prompt, task] # two messages to start
loop:
step()
if the last message is "exit" -> done
step():
execute(ask())
ask():
if the step limit is reached -> stop
if the cost budget is reached -> stop
if the time limit is reached -> stop
answer = model(messages) # MODEL CALL
cost += answer.cost
append_to_messages(answer)
execute(answer):
output = environment.run(answer.actions) # EXECUTION
append_to_messages(observation(output)) # OBSERVATION goes back
Four things show up here, and all four matter.
First: the history is linear. Every step appends its result to the previous ones. Nothing disappears, nothing compresses itself — the context grows monotonically.
Second: there are three stopping conditions, and every one of them is called a limit. Step count, cost, time. The default cost budget of this system is three dollars per task — the most concrete published number there is on how much a serious team is willing to spend on a single agent run.
Third: there is a fourth exit as well — after three consecutive formatting errors the system gives up. It does not keep trying forever.
Fourth, added by the system prompt: the model has to issue exactly one command in every response, and has to put a thought in front of it explaining why. This "thought–action" pair is not the model's natural behaviour. It has to be forced.
One step, up close
It is worth looking at what happens on the wire during a single tool call. The normative example in the observability standard describes exactly this:
agent run starts
|
+-- model call <- "what should I do?"
|
+-- tool call (client) <- the request goes out
| |
| +-- tool (server) <- the operation actually runs
|
+-- model call <- "I got this, now what?"
One tool call means two model calls. One before, one after. This is the single most important thing to know about the cost of agents — and the reason a ten-step task calls the model twenty times rather than ten.
Alongside the call, the standard also records the name of the tool, the arguments it was called with, what came back, and which protocol version was in use. If you log that, the run can still be reconstructed a month later — if you do not, it cannot.
What does this come to in numbers?
Concrete, published figures. Where a number comes from a vendor, I say so, because neither sample size nor measurement protocol was given for any of them.
- By one model developer's own measurement, agents consume roughly four times as many tokens as a conversation, and multi-agent systems fifteen times as many.
- Their own scaling guidance: for a simple question, one agent and 3–10 tool calls; for a comparison, 2–4 subagents with 10–15 calls each; for complex research, more than ten subagents.
- A single subagent typically burns tens of thousands of tokens, and returns a compressed summary of 1,000–2,000 tokens out of that.
- And the three-dollar budget above: at today's prices that is on the order of 1.5 million input or 300,000 output tokens. That is what fits into one run.
Where the name comes from: the ReAct pattern
Alternating thought and action is not a 2025 invention. The paper describing the pattern appeared in October 2022, and in conference proceedings in 2023.
The idea is simple: the model does not separate thinking from acting, it alternates between them. The thought revises the plan, the action pulls information from an outside source, and the two improve each other.
The results measured back then: 34 percentage points of absolute improvement on a household simulation task set and 10 percentage points on a web shopping task over the earlier methods — the latter with a mere one or two examples in the prompt.
The pattern is still alive: several no-code automation tools ship a separate "ReAct agent" node.
What can an agent not do?
This is the most important section, and there is a measurement behind every claim.
It is not consistent. There is a benchmark that measures not whether a task succeeds once, but whether it succeeds eight times out of eight. When it came out in 2024, the best systems of the day scored below 50% even on a single attempt, and below 25% eight times out of eight.
It has improved a lot since — but we do not know by how much. Today's leaders reach 85–88% on a single attempt on that same task set. The eight-out-of-eight figure, however, nobody publishes today. So capability has demonstrably grown, while there is no public data on consistency — and consistency is exactly what decides whether you can leave something unsupervised.
Where the task is new, it is still around 50%. On a harder, knowledge-retrieval variant of the same benchmark family, the best systems land between 48% and 55%.
Task length is the decisive factor. According to a March 2025 measurement, models solve almost 100% of what takes a human less than four minutes, and under 10% of what takes more than four hours. The same measurement also found that the task length models can handle roughly doubles every seven months — a trend estimate from March 2025, not a fact about today.
And one mechanical reason why a long run degrades. Because the loop appends the previous state at every step, the input grows monotonically. And according to a study across 18 models, output becomes steadily less reliable as the input gets longer — with even a single distracting item already hurting the result compared to the clean case. Which is why "let it run a bit longer" is not the answer.
One last, very tangible limit, from the prompt of that same production agent: directory changes and environment variables do not survive, because every operation runs in a fresh subshell. The agent only "remembers" what was written back to it as text.
Common misconceptions
- "The agent thinks, then it acts." It alternates between the two, and the thought has to be forced in the prompt — left to itself, it does not answer that way.
- "If I let it run longer, sooner or later it will solve it." A longer run means a longer input, and a longer input measurably means a less reliable output.
- "One step is one model call." It is two: one before the operation, one after the result.
- "If it solved it once, it can solve it." Consistency is a separate metric, and that is precisely where today's public data is missing.
- "The agent remembers what it did." Only what sits there in the text. Whatever happened in the environment and never came back as text does not exist for it.
- "A chatbot is an agent too, only smarter." The loop, the tool use and the feedback from the environment are the difference — not the size of the model.
What to take away
An agent is a loop with three stopping conditions and a monotonically growing text. If you keep that one sentence, the rest follows from it: why it is expensive, why it degrades on a long task, and why it needs limits.
The practical order that comes out of it: give it limits (steps, cost, time), give it tools instead of letting it work from its own knowledge, and log every step it takes, because afterwards that is the only thing the run can be reconstructed from.
How step count and reliability hang together — and why a twenty-step task fails even at 98% per-step accuracy — is worked through with the arithmetic in the agent reliability math post; the mechanism of the tool call itself gets its own walkthrough in the tool use and function calling post, field by field.