Free tools
Check what your robots.txt lets search engines read
Enter your domain. Lastmod reads your robots.txt the way a crawler does and tells you whether Googlebot and the AI crawlers can get in.
Free check
What does a search engine see on your homepage?
Enter your domain. We read your robots.txt, your sitemap and your homepage, and show what can be improved technically.
What we look at
- Your robots.txtAre search engines and AI crawlers allowed to read your site?
- Your sitemapCan search engines find your pages?
- Your homepageTitle, meta description, canonical and noindex.
One page, without signing in. A scan of all your pages, with your Search Console data, is part of the trial.
What it reads
- The rules for Googlebot
- Your own Googlebot group if you have one, otherwise the rules under User-agent: *. The check says whether the whole site is blocked.
- The AI crawlers, per provider
- For each provider, whether its search crawler and its training crawler are allowed or blocked, read from the same file.
- The sitemap line
- The sitemap address your robots.txt names, which is where the check looks first for your sitemap.
How to read the result
- Googlebot blocked
- A rule like Disallow: / under Googlebot or under *, with nothing more specific, shuts Google out of the whole site. This is the one result to fix first.
- Partly blocked
- Some paths are closed and others are open. That is normal for admin areas and internal search; check that no section you want found is among them.
- No robots.txt
- A missing file (a 404) means everything may be read. A file that cannot be reached at all tells the check nothing, so it does not report a block.
Common mistakes
- Disallow: / left over from staging
- A development rule that goes live with the site. The most expensive single line in SEO, and easy to miss because the site itself still loads.
- Using robots.txt to keep a page out of search
- A blocked URL can still appear in results without a description. To keep a page out, use noindex, and leave the page readable so the crawler can see it.
- The file in the wrong place
- robots.txt works only at the root of a host: example.com/robots.txt. Every subdomain needs its own.
What it cannot see
The check reads your robots.txt, not your CDN or firewall. A block set there is not visible to it.
Questions
How do I check my robots.txt?
Open yourdomain.com/robots.txt in a browser to read the file, or enter your domain here to see what it means for Googlebot and the AI crawlers.
Does robots.txt remove a page from Google?
No. It controls what a crawler may fetch, not what is indexed. A blocked page can stay in the results; use a noindex instruction to keep a page out.
Is the check really free?
Yes. It needs no account and nothing is installed. It reads one page and your public files, as Lastmod-Crawler.
Want the whole site checked?
Start the free trial: Lastmod scans all your pages and ranks what it finds.
30-day free trial · no credit card · database and hosting in the EU