Skip to content
Lastmod

Glossary

Crawl budget

Crawl budget is the number of pages a search engine is willing and able to fetch from your site in a given period, set by how fast your server responds and how much your pages seem worth fetching.

For most sites it is not a bottleneck. Google says crawl budget is mainly a concern for very large sites, with hundreds of thousands of pages or more, and for sites whose content changes many times a day. A site of a few hundred or a few thousand pages is usually crawled in full.

When it does matter, the waste is the usual suspect: many near-identical URLs from filters and parameters, redirect chains, error pages and sitemap entries that lead nowhere. Cleaning those up lets the crawler spend its visits on pages you want found.

If a new page is slow to appear in search, crawl budget is rarely the first thing to check. Start with whether the page is linked, in the sitemap, and not blocked or marked noindex.

In Lastmod

Lastmod does not measure your crawl budget or how often Googlebot visits. Its crawl does flag things that waste crawls: sitemap URLs that redirect or fail, and internal links that go through a redirect.

See it on your own site

Start the 30-day free trial. Technical checks run as soon as Lastmod has access to your domain and can read your sitemap.

30-day free trial · no credit card · database and hosting in the EU