The mechanism: who does what?
Four steps, and the third one is the point.
1. The developer describes the tool. Three fields: name, description and input schema in JSON Schema form. The description is not documentation for your team — it is the model's only clue about when to use the tool.
2. The model fills in the form. What comes back in the response is not text but a tool-call block: which tool it wants, with which arguments, and an identifier the result can later be matched against. The stop reason of the response switches over to "tool use" as well.
3. Your code executes it. The vendor documentation puts it literally: client-side tools run in your own application — the model replies with the tool-call block, your code performs the operation, and sends the result back.
There is one exception: server-side tools — web search, web fetch, code execution — run on the provider's infrastructure, where you execute nothing, you only receive the result.
4. The result goes back into the conversation. As a result block, paired with the identifier of the original call. The model continues from there.
And from this follows the biggest cost item in agents: one tool call means two model calls. One before, one after.
Where does it go wrong?
There is an academic leaderboard measuring function calling specifically, and its version history says more than the current ranking: simple calls → live data → multi-turn and multi-step calls → agentic evaluation.
Four failure patterns, measured, on a task set of a hundred multi-step questions:
| Failure pattern | What the model does |
|---|---|
| Tool avoidance | it has a search tool and still answers from its own knowledge — with a stale or incomplete result |
| Wrong assumption | it assumes an event has not happened yet, so it never starts the search |
| Bad decomposition | it does not break the multi-step question into steps, it tries one all-encompassing search instead |
| Misreading | it finds the right page and misinterprets its content |
A side finding from the same measurement: without tool use, accuracy drops dramatically. So the tool is worth a great deal — but all four failures above happen to somebody who has a tool.
And a non-trivial result that surprises a lot of people. The same leaderboard separately measured format sensitivity across 39 models: same tool, same model, only the documentation format differs.
- For the format of the function documentation there is a consistent trend across nearly every model: JSON is best, XML weaker, Python worst.
- For the return format, Python and JSON both beat every XML variant — and smaller models are more sensitive to it.
- Without tool-call tags the models perform somewhat better than with them.
- To the format and style of the prompt, on the other hand, models are largely indifferent.
The extremes are telling: one model under test dropped to near zero performance purely because JSON format was demanded of it, and another did the same over tool-call tags.
The practical lesson: if it is not working, swap the format before you swap the model.
How many tools fit in a prompt?
Two vendors give two different numbers here, and neither publishes a methodology.
| Source | Recommendation | Nature |
|---|---|---|
| One vendor | under 20 at the start of a round | "loose suggestion", with no reasoning |
| The other | above 30–50 the choice degrades | a claim, with no measurement data |
What there is a concrete number for: a typical setup wiring several servers together burns roughly 55,000 tokens on tool definitions alone, before the model does anything. According to the vendor, a tool-search approach cuts that by more than 85%, because it loads only the 3–5 tools relevant to the question.
An even sharper figure from the same source, with a code-execution approach: from 150,000 tokens down to 2,000 — a saving of 98.7%.
But for the question of exactly where the breaking point is, there is no independent measurement. What we do know: more context measurably means a less reliable output — but no study measured that as a function of tool count. This is a genuine open question.
In practice one rule follows from it: the question should not be how many tools you can wire up, but how many the model sees on a given request.
What is not guaranteed
Three things the documentation explicitly does not promise.
You do not know how many calls are coming. One vendor states literally that a response may contain zero, one or more calls, and that your code is best off being ready for all of them. Parallel calling is on by default, but can be switched off.
Schema compliance is not factual correctness. There is a mode that forces the call to match the schema. That is useful — but it does not guarantee that the model picked the right tool or the right values. All four failure patterns above are schema-valid, factually wrong calls.
The error has to be written back into the conversation. The result block has a dedicated error flag — that is what the model learns from for the next step. If you just swallow the error, the model does not know it got it wrong. A production agent has a backstop for this too: after three consecutive formatting errors the run stops.
How much more expensive is a round with a tool call?
Two items are added before you do anything at all.
The first is the system prompt surcharge. Enabling tool use automatically adds a system prompt whose size is published per model — for today's models roughly between 280 and 590 tokens, depending on whether the model may choose freely or is required to call a tool. This lands before the tool definitions even start.
The second is the definitions themselves. They are billed as input tokens, on every single call.
Own calculation. Take a mid-priced model ($2 per million input tokens, $10 per million output) and compare two cases:
| Plain answer | One round with a tool call | |
|---|---|---|
| Number of model calls | 1 | 2 |
| Total input tokens | 500 | 2,688 |
| Total output tokens | 150 | 230 |
| Cost | $0.0025 | $0.0077 |
The multiplier: 3.07×. Over a thousand rounds that is $7.68 instead of $2.50.
(Own calculation. The system prompt surcharge is a measured, vendor-published figure; the size of the tool definition and the tool result are my own estimates, and in reality they vary widely. The calculation contains no prompt cache.)
Two things that shift this.
Upwards: with N tool calls there are N+1 model calls, and every call carries the entire history so far. So the multiplier does not stop at three — on a long run it keeps getting worse.
Downwards: the prompt cache is the single most important optimisation. On a cache hit, input tokens go at a tenth of the price — and since most of the second call's input is word for word the same as the first one's, it bites exactly where it counts most.
A cross-check against the own calculation: one vendor's own measurement says agents consume roughly four times as many tokens as a conversation. The 3.07× applies to a single call, and it sits comfortably in that same range — the two numbers arrived by independent routes.
Common misconceptions
- "The model calls the API." It does not. It fills in the form, your code makes the call. The exception is the server-side tool running at the provider.
- "If the schema is strict, the call will be correct." The schema forces the shape, not the content. The overwhelming majority of measured failures are schema-valid.
- "The tool description is documentation." The tool description is a prompt. It is the one thing the model decides from when to use the tool.
- "I will wire up everything and let it pick." A setup joining several servers can burn 55,000 tokens on definitions alone, before any work happens. What counts is not the number of tools wired up but the number visible to the model.
- "One call always comes back." Zero, one or more. The code has to be ready for each case.
- "Only the model matters." The same model gives a measurably different hit rate with a different documentation format — JSON is consistently better than XML or Python.
What to take away
Function calling is not a call, it is a proposal. The model says what it would like; execution, error handling and permissions stay on your side. That is good news: this is where you can intervene.
The three things that help most in practice: a good tool description (because it is a prompt), JSON format (because models are measurably better at it), and prompt cache (because most of the input in a tool-call round repeats).
How this turns into a full agent loop is walked through end to end in the what an agent actually does post; and the standard that tools are usually wired up with these days gets its overview in the MCP post, together with its security caveats.