Skip to content
Lastmod

Lastmod-Crawler

Last updated: 2026-10-15

Lastmod-Crawler is the bot of Lastmod, an SEO platform. If you found it in your server logs, a Lastmod customer has added your website to their account and either verified that they own it or connected a Google Search Console account that has access to it.

Why it visits your site

Lastmod customers use the crawler to check the pages of their own website: titles, descriptions, links and other technical SEO signals. The crawler only fetches websites for which the customer has shown ownership, through a DNS TXT record or as owner of the site in Google Search Console, or has access to the site in Google Search Console (a Google account with access to the whole domain). Access is not ownership: without proven ownership the crawler stays on the website’s own two addresses, the domain and its www version, and does not visit any other subdomain. It does not crawl the open web and does not follow links to other domains. It also never visits one host more than once at a time, however many customers have added it.

How to recognise it

Every request carries this User-Agent header: Lastmod-Crawler/1.0 (+https://www.lastmod.ai/en/crawler)

The product token is Lastmod-Crawler. We do not publish an IP list: the crawler runs on shared cloud infrastructure without fixed addresses, so identify it by the User-Agent.

robots.txt

The crawler fetches /robots.txt once per crawl and follows it. It looks for a group with User-agent: Lastmod-Crawler and falls back to User-agent: * when there is none.

To keep the crawler out of your whole site, add this:

User-agent: Lastmod-Crawler
Disallow: /
  • Allow and Disallow rules are respected, including * and $ patterns.
  • Crawl-delay is respected up to a maximum of 10 seconds.
  • Sitemap lines in robots.txt are read to find your pages.
  • If robots.txt cannot be reached (server error or timeout), the crawler skips those pages instead of guessing; they are reported as skipped, not as failed. A 4xx response, such as 404, counts as no restrictions. Single-page checks you start yourself, such as a fix check, follow robots.txt too, and so does every redirect: a redirect to a disallowed path is not followed.

Pace

The crawler makes one request at a time per website and pauses between requests. The pause follows how quickly your server responds: a slower server gets a longer pause, and the pause is longer again after an error. Each crawl is limited in the number of pages and in total duration.

Contact

Does the crawler cause problems, or do you not want it on your site? Write to support@lastmod.ai with the domain and, if you can, a few log lines. We will respond as soon as we can.