Opportunity library
AI bot access
Your robots.txt may block the crawlers of AI assistants, which keeps your pages out of what they can read.
- Applies to
- The whole site
- How much work
- A quick fix
- Ends in
- Look at it and decide
How Lastmod finds it
Lastmod reads your robots.txt per AI crawler, for example GPTBot, ClaudeBot and PerplexityBot, and says which are blocked.
What you do
Decide per crawler whether you want it, then change the matching rule. Either choice can be recorded as a decision.
How the work is followed up
There is nothing on the site to check. You record what you decided and why, and Lastmod keeps it with the date.
More about this problem
Other types in this group
See which of these apply to your site
Start the 30-day free trial. Lastmod runs the checks and shows what it found, with the evidence.
30-day free trial · no credit card · database and hosting in the EU