The short answer
AI companies run three kinds of bot: search bots that index pages so an assistant can cite them, training bots that collect pages to train models, and fetchers that load a page when a user asks. Blocking a search bot can remove you from that assistant's answers. Blocking a training bot is a separate choice. Robots.txt is a request, not a lock.
Three kinds of AI bot
The same company often runs several bots with different jobs, and each has its own name in robots.txt. Treating them as one group is the most common mistake.
- Search bots index pages for an AI search or answer product. OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic) and PerplexityBot are examples.
- Training bots collect pages to train or ground models. GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl) and Amazonbot are examples. Google-Extended and Applebot-Extended are control tokens: they have no crawler of their own and only state whether your pages may be used for AI training.
- User-triggered fetchers load one page because a person asked an assistant about it. ChatGPT-User and Perplexity-User are examples. Several providers say these may not follow robots.txt.
Decide per kind, not per company
If you want assistants to name and link to your site, the search bots must be allowed. Blocking one can take you out of that assistant's answers. That is a real cost for a site that wants to be found.
Whether to allow training bots is a separate decision about your content and your terms, and there is no right answer for every site. A publisher that sells its text may block them. A company that wants its product explained everywhere may not. What matters is that it is a decision.
A search engine's own control token, such as Google-Extended, governs AI training use and, according to Google, does not affect ranking in Google Search. Read the provider's own documentation before relying on that for any bot, and expect it to change.
How the rules look
Each group starts with a User-agent line naming the bot, followed by Allow and Disallow lines. A bot follows the most specific group that names it, and falls back to the group for all bots when none does.
To allow search bots and block a training bot, write one group per bot. To keep a section private from everyone, put the rule in the group for all bots. Names are not case sensitive, but spelling matters: a rule for GPT-Bot does nothing for GPTBot.
- User-agent: OAI-SearchBot, then Allow: /
- User-agent: GPTBot, then Disallow: /
- User-agent: *, then Disallow: /admin/
Where the evidence stops
Robots.txt is a request
A bot that follows the rules reads robots.txt before it fetches anything. Nothing forces a bot to do that, and nothing in the file keeps a page private. If a page must not be read, put it behind a login.
A blocked page can also still be known: if other pages link to it, the address can be listed without its content. To keep a page out of search results, the page must be reachable and carry a noindex directive. A robots.txt block does the opposite.
The accidents to look for
Most blocks that hurt are not decisions. A broad rule written for one bot that applies to all of them, a staging file copied to production, a security plugin or hosting setting that adds a block for AI bots without telling you, a Disallow: / left from a launch.
Check the file after every release, and check what the bots actually see: the live file at the address /robots.txt, not the version in your repository.
A short checklist
Before you change anything:
- Open yoursite.com/robots.txt and read it, including groups for specific bots.
- Write down which kinds of bot you want to allow and why.
- Make sure no rule for all bots blocks the sections you want assistants to cite.
- Test the live file after the change, and again after the next deploy.
In Lastmod
Lastmod reads your robots.txt per AI crawler and reports which are blocked, so a block is visible as a decision. It reads the file and nothing more: it does not judge whether you should block a training bot, and it makes no claim about what allowing a bot does for your visibility. You can try it on any site with the free AI crawler check.