Free tools
Check which AI crawlers your robots.txt blocks
Enter your domain and see, per provider, whether the AI search crawler and the training crawler can read your site.
Free check
What does a search engine see on your homepage?
Enter your domain. We read your robots.txt, your sitemap and your homepage, and show what can be improved technically.
What we look at
- Your robots.txtAre search engines and AI crawlers allowed to read your site?
- Your sitemapCan search engines find your pages?
- Your homepageTitle, meta description, canonical and noindex.
One page, without signing in. A scan of all your pages, with your Search Console data, is part of the trial.
What it reads
- Search and training, apart
- A provider often runs one crawler for its AI search and another for model training, for example OAI-SearchBot and GPTBot at OpenAI, or Claude-SearchBot and ClaudeBot at Anthropic. The check keeps them apart.
- Allowed, blocked or partly
- The outcome per crawler, from the rules in your robots.txt. Where there is no group for a crawler, the rules under User-agent: * apply.
- Only when it can judge
- If your robots.txt could not be read, the check shows no AI-crawler result rather than a guess.
How to read the result
- Training crawler blocked
- A choice, not a mistake. Many sites block training and stay open to search. The check reports it without calling it a problem.
- Search crawler blocked
- Worth a second look. If the AI search crawler of a provider cannot read your pages, that provider's answers have a harder time citing you.
- Everything allowed
- Nothing in your robots.txt keeps these crawlers out. Whether they visit, and whether you are cited, is a separate question that robots.txt does not answer.
Common mistakes
- Blocking with User-agent: * by accident
- A broad rule written for one bot closes the door on all of them, search crawlers included.
- Treating robots.txt as a lock
- It is a request that well-behaved crawlers follow. It does not secure anything, and some providers say that certain fetchers started by a user do not follow it.
- Assuming a block hides you from AI answers
- A blocked crawler cannot read your pages, but an assistant can still name your brand from other sources.
What it cannot see
The check reads robots.txt only. It does not see whether a crawler actually visits you, and a block set in a CDN or firewall is not something it can see.
Questions
Should I block GPTBot?
That is your decision. Blocking a training crawler keeps your pages out of model training; blocking a search crawler can keep you out of that provider's AI search answers. The check shows which is which.
What is the difference between GPTBot and OAI-SearchBot?
OpenAI lists GPTBot as the crawler for training its models and OAI-SearchBot as the one for search results in ChatGPT. You can set different rules for each.
Does this tell me if I am cited by AI assistants?
No. It tells you whether the crawlers are allowed in. Whether an assistant mentions you is a different measurement.
Want the whole site checked?
Start the free trial: Lastmod scans all your pages and ranks what it finds.
30-day free trial · no credit card · database and hosting in the EU