SaaS SEO · Technical SEO

    SEO log file analysis: what Googlebot actually does on your site

    Franck Gordi, SaaS SEO and GEO consultant

    Franck Gordi

    9 min read

    Contact me
    SEO log file analysis

    Requested URL

    Response code

    User agent

    SEO log file analysis is the review of server logs to observe what search engine crawlers actually requested on a site, how often, and with which response. This article is for whoever runs SEO on a B2B SaaS and wonders whether that work is worth its cost. Direct answer: logs are the only source describing observed crawler behaviour, where a crawl only describes a theoretical state. On a site of a few hundred pages with no generated URLs, that difference often changes nothing. On a faceted site, on a site mid-migration, or on a site that wants to verify AI crawler access, it changes everything.

    Key takeaways

    • A log file records the requests received by the server: date, URL, response code, user agent, IP address. Nothing else, and that is enough.
    • A crawl answers the question of what is reachable, logs answer what was actually requested. The two are cross-referenced, never substituted.
    • User agents can be spoofed: a visit is only attributed to Googlebot after a reverse DNS lookup, a procedure documented by Google Search Central.
    • Google states that crawl budget management is not a concern for most sites. Log analysis is justified by an observed problem, not by principle.
    • Logs verify a precondition of AI visibility: if GPTBot or PerplexityBot cannot access a page, no citation is possible.

    1. What a log shows and no other tool sees

    A log file is the journal of requests received by the server. Each line matches one demand: a visitor or a crawler asked for an address, and the server answered with a code. That journal already exists on any installation. It does not need to be installed, only retrieved and retained.

    What a line contains

    Four fields do all the work

    Date and time

    Crawl rhythm, day by day, section by section.

    Use: Spot a drop in frequency or a spike after a release.

    Requested URL

    What the crawler is really asking for, parameters included.

    Use: See crawled addresses nobody ever decided to publish.

    Response code

    What the server answered at that precise moment.

    Use: Tell a one-off error apart from an installed problem.

    User agent

    Which crawler asked for the page, classic engine or AI engine.

    Use: Confirm crawler visits, to be verified against the IP address.

    The value of this source comes down to one point: it is descriptive. Search Console provides aggregated crawl statistics that are genuinely useful, but it does not deliver the full list of requested addresses, line by line. A crawling tool simulates an ideal robot following links from an entry point. Logs record real crawlers, which sometimes land on addresses nobody ever linked to.

    The practical difference with a crawl

    A crawl answers the reachability question: is this page accessible, at what click depth, with which tags. That reading is structural, and it is exactly the one described in internal linking as a structure of priorities. Logs answer a different question: among those reachable pages, which ones are actually requested, and at what rhythm. A gap between the two is where the analysis begins.

    2. How to read logs, step by step

    The order matters, because each step determines the reliability of the next. An analysis run on unauthenticated lines produces conclusions about crawlers that never were.

    The method

    Four steps, in this order

    01Pull a continuous window and authenticate the crawlersThirty days of logs with no gaps, then verification through reverse DNS lookups. A declared user agent proves nothing: half the lines attributed to search engines can come from somewhere else.
    02Cross-reference logs with your published URLsOn one side the sitemap and the crawl, on the other what was actually requested. The intersection produces two lists that exist nowhere else: published pages never visited, and visited addresses never published.
    03Group by section, not by URLA single URL says nothing. Volumes are read by template and by directory: blog, service pages, documentation, generated pages. That is the level at which crawl distribution becomes a decision.
    04Compare crawling against page valuePages that carry conversion should be visited at least as often as secondary pages. A lasting imbalance is an internal structure problem, not a server problem.

    Step 01 is the one most often skipped. Google documents a two-part verification: a reverse DNS lookup on the IP address, which must return a name in googlebot.com or google.com, then a forward lookup on that name, which must return the original address. Without that check, part of the volume attributed to the search engine can come from a third-party tool or a content scraper.

    Group before interpreting

    A useful analysis is not read URL by URL but by template and by directory. The goal is to answer a distribution question: what share of crawling goes to the pages that carry qualified demand, and what share goes to the rest. It is the same prioritisation logic described in B2B SEO driven by qualified demand, applied this time to crawler behaviour.

    3. Telling a signal from noise

    Crawling is irregular by nature. A page visited twice this month and five times last month says nothing. What can be read are stable trends across sets of pages, and anomalies that repeat.

    A real signal

    • Service or documentation pages that are published and never requested across the whole window.
    • A large share of crawling absorbed by parameter URLs nobody ever wanted.
    • Recurring server error codes on the same template, at the same hours.
    • Redirect chains still being requested weeks after a migration.

    Noise

    • A page visited three times instead of five from one month to the next.
    • An isolated crawl spike after a release or a sitemap update.
    • One-off errors on URLs that were deliberately removed.
    • Lines attributed to a search engine without any IP verification.

    The most frequent case on a SaaS site: parameter addresses, generated by a filter or an internal search engine, absorbing a significant share of requests. The reflex is often to block them. The better first move is to remove the cause when it is internal, meaning the links that manufacture them. Blocking leaves the structure intact, fixing the structure solves the problem at its source.

    The case of generated pages

    When a site produces pages from a template, logs become the control instrument for that batch: how many of those pages are actually crawled, at what rhythm, and with what result. That is the control most often missing from the projects described in programmatic SEO and what produces empty pages.

    4. When this analysis is justified, and when it is not

    Google is explicit on this point in its documentation on crawl budget: most sites do not need to worry about it. A log analysis sold by default, on a site with no generated URLs and no incident, is a deliverable that will confirm the obvious.

    Case 1

    Many generated URLs

    Facets, filters, tracking parameters, pages created by an internal search engine. This is where crawling scatters fastest, and where logs are the only available measurement.

    Case 2

    Right after a migration

    You check that new URLs are actually requested, that old ones return a clean response in a single step, and that nothing is blocked by a leftover staging rule.

    Case 3

    AI crawler control

    Logs confirm whether generative engine crawlers reach your pages or receive a refusal, which no citation tracking tool can establish from the outside.

    Case 4

    A site of a few hundred pages

    With no generated URLs and no migration in progress, reading logs mostly confirms what Search Console already shows. The effort pays off better elsewhere, on content and internal structure.

    Case 2 deserves emphasis, because that is where the ratio between effort and information is most favourable. After a switch, logs answer within hours what Search Console will take weeks to document: are the new addresses being requested, and do the old ones return the right response in a single step. That is the control described in the SEO migration plan, observed from the server side.

    5. What logs bring to AI visibility

    Generative engines cite pages they were able to read. The relevant agents are identifiable in the server log: GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity. Their presence, or their absence, is observed line by line.

    Two checks are enough, and they cost almost nothing. First, do those agents reach the pages you want cited. Second, what response do they receive: a 200 code, or a refusal returned by a firewall or a CDN protection blocking AI crawlers without anyone having decided to. This is a precondition check, not a performance measure: the citation rate is measured elsewhere, as explained in AI visibility tracking tools. It is also part of the technical scope of an LLM visibility audit.

    Frequently asked questions

    SEO log file analysis is the review of server logs to observe what search engine crawlers actually requested on a site. Each line records one incoming request: the date, the URL called, the response code returned, the user agent and the IP address. It is the only source that describes observed crawler behaviour, whereas a crawling tool only describes what a crawler could find.

    A crawl simulates a robot: it starts from one URL, follows links and produces the theoretical state of the site. Logs record the actual visits of real crawlers. A crawl answers the question of what is reachable, logs answer what was requested, how often, and with which response. The two do not replace each other, they are cross-referenced.

    Thirty consecutive days is a reasonable baseline for a site of a few hundred to a few thousand URLs, because crawling is irregular from one day to the next. Over a shorter window, a rarely crawled section can look ignored when it was simply visited the week before. After a migration or a redesign, a shorter window is enough to check that new URLs are being discovered.

    The user agent is easy to spoof, so it proves nothing on its own. Verification runs a reverse DNS lookup on the IP address, which must resolve to a name in googlebot.com or google.com, then a forward lookup on that name, which must return the original address. Google documents this procedure and also publishes its crawler IP ranges.

    On a site with fewer than a few thousand URLs, no facets and no parameters, wasted crawling is rarely the problem. Google itself states that crawl budget management is not a concern for most sites. Logs still serve two precise purposes: controlling a migration, and checking whether generative engine crawlers reach your pages.

    Three levers, in this order. Remove the cause when it is internal, meaning the links that generate those addresses. Block crawling of worthless URL patterns in the robots.txt file. Return a clean response for whatever should disappear, instead of leaving redirect chains or soft error pages that keep being requested.

    They verify a precondition, not a result. Logs show whether agents such as GPTBot, OAI-SearchBot, PerplexityBot or ClaudeBot actually access your pages, and with which response code. A page never reached by those agents cannot be cited. The citation rate itself is measured on a panel of prompts, not in the server log.

    Conclusion

    Log analysis is not a growth lever, it is a measurement instrument. It answers factual questions nothing else settles: is this page seen, does this section absorb crawling without producing anything, does this crawler actually get a response. On a SaaS site with generated pages, a redesign in progress or a doubt about AI crawler access, those answers justify the setup. Elsewhere, the effort produces more value on content and internal structure.

    If you are unsure whether this work is relevant for your site, we can look at your situation in a first call, and you leave with a clear view of what should come first.

    Contact me

    Sources

    About the author

    Portrait of Franck Gordi, independent SEO and GEO consultant for B2B SaaS

    Franck Gordi

    Independent SEO & GEO consultant for post-funding B2B SaaS. 13+ years of experience, 824+ projects. He makes SaaS companies visible on Google and cited by AI engines, in French and in English.

    LinkedIn →