The four bots that are not the same
Most robots.txt files get written by someone pasting in a list off the internet. That turns out expensive here, because the large providers run several robots with different purposes, and those do very different things.
- Training bot. It collects material for training the model. Blocking it has no effect on whether you appear in the answers.
- Retrieval bot. This is the one that looks for sources while the answer is being written. Blocking it takes you out of the answers.
- User-initiated fetch. When somebody specifically supplies or asks for your page. Several providers note in their own documentation that this does not necessarily respect
robots.txt. - Classic search bot. The traditional index. This one has been around a long time and has to be handled separately.
The practical consequence: robots.txt is not something you set up "for AI", you set it up bot by bot. Anyone who blocks everything shuts out both the training and the appearance in answers — and the second of those takes away their visitors.
The block that takes you out of the answers
This is the most important sentence in the post, and it is not interpretation but a literal vendor statement: one large provider's documentation says that pages opted out of their retrieval bot do not appear in their search answers.
This is the point where one wrong robots.txt line can hurt immediately and measurably. So it is worth deciding not on principle but separately: training and appearing in answers are two different questions, and for most companies the answer to each is different. Plenty of people would rather not be used for training; almost everyone wants to appear in the answers.
The JavaScript nobody has fresh data on
The most common technical question: do the bots run JavaScript? The short answer: the classic search bot does, most AI bots documentably do not, or only to a limited extent.
The longer answer is more uncomfortable. The last serious log-based measurement of this is over a year and a half old, and no provider has said anything substantial about it since. So there is no fresh, reliable data here — only the old measurement and common sense. What to take away from that: if your content only appears after client-side JavaScript, you are taking a risk. It is not a hundred percent, it is a probability — but server-side rendering is cheaper than the uncertainty.
The structured data that does not help
This will surprise a lot of people. A 2026 study built on a comparative method examined almost two thousand pages and four thousand matched controls to see what adding structured data does for appearing in AI answers.
The result: at one surface minus 4.6 percent (statistically significant), and at the other two indistinguishable from zero. So it does not help — in one case it does harm. Incidentally, the largest search engine's own official documentation confirms this: there is no special structured data you need to add.
None of which makes structured data useless: the traditional search engine needs it for richer result displays, and products need it for shopping surfaces. Just do not do it for the sake of AI citations.
The llms.txt nobody requests
llms.txt is the idea of putting out a file, modelled on robots.txt, that summarises your site for language models. It became widespread advice, and plenty of tools generate one.
The reality: according to an analysis of the logs of more than a hundred and thirty thousand domains, 97 percent of these files received zero requests. Where there was traffic, only a negligible share of it came from AI retrieval bots. Not one large provider supports it officially. There is a quiet irony in it: one provider does use such a file for its own developer documentation — for coding agents, not for search.
That does not mean it is forbidden. It means do not spend more than five minutes on it, and do not expect results from it.
The paid access being built right now
A new layer is starting to take shape: content providers can charge robots for access. The best known scheme of this kind was still in closed beta in the summer of 2026, with a token unit price, and the wider service attached to it runs on a waiting list.
What is live: from September 2026 that same provider blocks AI crawlers by default on part of the sites it serves, unless the owner says otherwise. This matters: something like this may be active on your site without you ever having switched it on. If you want AI traffic, check this with your provider.
What to do today
1. OPEN your robots.txt and go through it bot by bot.
The training bot separately, the retrieval bot separately.
If the retrieval bot is blocked, unblock it now.
2. CHECK whether your provider (CDN, hosting, shop platform)
blocks on its own. From September 2026 that can be the default.
3. LOOK AT your own site with JavaScript switched off.
Turn JS off in the browser. What you cannot see, the bot cannot.
4. DO NOT WRITE an llms.txt in the hope that it helps.
If you do it, make it five minutes, not a project.
5. DO STRUCTURED DATA for the classic search engine,
not for AI citations. It does not work for that.
Common misconceptions
- "AI bots should be handled as one group." Blocking the training bot and blocking the retrieval bot produce completely different outcomes. One protects your content, the other takes away your visitors.
- "Structured data helps with AI citations." In a large-sample study it did not help, and at one surface it hurt — and the search engine's official docs say the same.
- "llms.txt is the new robots.txt." Nobody requested 97 percent of those files.
- "If I have not blocked anything, everything is fine." Your hosting provider or your CDN may be blocking without your knowledge.
What to take away
With machine readability, access is the point, not markup. One wrong line in robots.txt does more harm than any structured data can do good — and your provider's defaults have a say as well. Go through it bot by bot, look at it with JavaScript off, and do not spend time on files nobody requests. What counts on the content side is set out by the factors behind citation post, with the studies, and on measuring the effect the tracking your brand visibility post gives the metrics.