Crawl budget: does your SaaS really have a crawling problem?

When does crawl budget become a real issue?

When useful pages are waiting to be crawled and a large inventory, URL multiplication or a server constraint may explain that wait. Page count is only one clue. Discovery time for priority content and the nature of requested URLs matter more when deciding whether work is needed.

Google's crawl budget guide mainly targets three situations: more than one million unique pages with moderately frequent changes, more than 10,000 unique pages changing daily, or a significant proportion of URLs marked “Discovered, currently not indexed.” Google says these numbers are rough guides, not exact thresholds.

A SaaS may have a small marketing site and a vast public inventory: versioned documentation, an integration directory, templates, profiles or filter combinations. Assess the inventory available to crawlers, not just pages visible in the menu. Conversely, an authenticated application is not a public catalogue that needs crawling.

Capacity and demand: two different levers

Crawl capacity describes what Google can retrieve without overloading the host. Crawl demand describes what its crawlers want to retrieve. Crawl budget results from both, not from a fixed daily quota you can set in a tool.

An available server does not create demand

Slow responses, 5xx errors and 429 responses can reduce crawl capacity. A more stable server can remove that constraint. But if Google sees little content worth revisiting, frequency will not automatically increase. Oversized infrastructure does not turn low-value variants into priority pages.

The perceived inventory matters as much as the intended inventory

Google knows the URLs it discovers, not only those your team considers published. Sort parameters, old versions and generated links can expand that inventory. The useful question becomes: which URLs provide distinct information, and which merely multiply paths to the same information?

Google defines a site by hostname here. A marketing site and a documentation subdomain may therefore have separate crawl budgets. Analyse their behaviour separately before drawing a conclusion about the brand as a whole. This is not a reason to move pages artificially onto new subdomains.

Discovered or crawled: where does the path stop?

The page indexing report shows different states that do not call for the same action. “Discovered, currently not indexed” means Google knows the URL but has not crawled it yet. “Crawled, currently not indexed” means Google retrieved it without adding it to the index for now. Neither label gives the complete cause by itself.

Start with a sample of pages prospects genuinely need to find: an integration page, use case, public documentation or product page. Inspect the URL, status, latest crawl when available, canonical and any restrictions. Compare pages using the same template, not a new page against an old popular resource.

The page indexing report locates the blockage. Crawl statistics can then reveal host degradation, longer response times or errors. Logs complete the picture when you need the exact URL families requested. They do not explain on their own why Google declines to index content.

If several useful page families remain blocked and the signals conflict, a technical SEO diagnosis of crawling and indexing helps select the right work before involving developers.

Reduce useless URLs without removing valuable ones

The first lever is the source of URLs, not robots.txt. If a template produces endless combinations, fix what generates or exposes them. Blocking those combinations without understanding their origin lets the system keep producing an inventory nobody controls.

Documentation, filters and programmatic pages

An old documentation version may still serve customers. A filter can answer a precise search. An integration page may have independent value. Do not classify a whole family as useless just because it contains a parameter or resembles another page.

For each family, distinguish user utility, search value and update needs. A page that must remain accessible but should not appear in Google needs different handling from a duplicate to consolidate or a permanently removed URL. That editorial decision comes before the technical rule.

If near-duplicate pages are the issue, a content strategy built around distinct intents will be more useful than simply increasing crawl frequency. More visits to copies do not give Google a reason to index them.

A canonical is not a crawl lock

A canonical helps consolidate similar versions; it does not stop Google requesting a variant. Keep internal links, sitemap URLs and preferred versions aligned. If links continually promote variants, the tag alone does not remove their discovery.

Can noindex control crawl budget?

Not as a stop button. Google must retrieve the page to read noindex. The rule removes content from the index, although deindexing may indirectly change crawl priorities. The objective matters: preventing a visit and preventing a result from appearing are not the same thing.

Google's noindex documentation is explicit: the rule must be accessible to the crawler. Blocking the same URL in robots.txt can prevent it from being read. According to the robots.txt documentation, a crawl-blocked URL may still appear in results if Google discovers it elsewhere.

A broad parameter rule may block valuable pages. A resource directory rule may prevent content from rendering. Test the intended scope and exceptions before release, then inspect representative URLs. A supposed saving in requests does not justify making commercial pages unreadable.

How do you know whether a fix worked?

The expected result is not “more crawling” but suitable crawling of pages that matter. Fewer requests to duplicates can be healthy. More requests to low-value URLs are not a success.

• Keep a stable group of priority pages with their real publication or modification dates.
• Track discovery and recrawling, then measure indexing separately.
• Compare errors and response times across comparable periods, noting infrastructure changes.
• Check excluded families to ensure rules did not affect valuable pages.

Do not alter sitemap dates to manufacture a crawler return. Google recommends a lastmod matching an actual change. A date edit is neither a content improvement nor evidence of freshness.

Do not promise mechanical reallocation either: Google's guide says released budget is not automatically reassigned to other pages. Reducing noise improves efficiency, it does not reserve visits for conversion pages.

Does more frequent crawling improve rankings?

No, not by itself. Google explicitly separates crawling from ranking. Fixing a retrieval problem removes an obstacle; it does not prove that the page answers the query better, deserves indexing or should outrank competitors.

The same caution applies to AI engine visibility: this article describes Google crawler behaviour. It cannot establish how every other crawler behaves or promise citations. A crawler visit and a recommendation to a prospect are two different observations.

When the source of the problem remains unclear, an SEO audit that prioritises bottlenecks prevents speed, content and crawling from being treated as one issue.

About the author

Franck Gordi

Independent SEO and GEO consultant for B2B SaaS. He helps software companies become visible on Google and cited by AI engines.

LinkedIn →