Definition
Log file analysis is reading your web server’s raw request records to see exactly which bots and visitors fetched which URLs, and when. Unlike analytics, which counts only sessions that ran a tracking script, a server log records every request, including AI crawlers that never execute JavaScript. It is your only first-party record of what a crawler fetched.
At a glance
- CategoriesTechnical
- Related fieldsAnalytics, Technical
- DifficultyAdvanced
At a glance
- A log line holds a timestamp, an IP address, the URL requested, the response code and the user agent.
- AI crawlers appear by user agent, so filtering for them takes one command and no tooling.
- A 200 proves the page was served, not that the content inside it was readable.
What a request line actually tells you
Every request writes a line. Three fields carry most of what matters: the user agent naming the client, the path it asked for, and the status code it got back.
Filter by user agent and the picture forms fast. GPTBot, ClaudeBot, PerplexityBot and CCBot each appear under their own name, so you can see which sections each one visits and how often.
Two patterns deserve attention. Crawlers hitting shallow pages repeatedly while never reaching your product or guide pages, usually an internal linking problem. And crawlers getting 404s or 403s, meaning a rule is blocking a client you meant to allow.
Logs record what left your server. They say nothing about whether the HTML held your content or an empty shell waiting on JavaScript.
Logs settle arguments analytics cannot
Someone says the assistants are ignoring your site. Analytics cannot settle it, because AI search traffic is a downstream effect with a broken referrer.
The log answers the upstream question. Either an AI crawler requested your pages or it did not, and that one fact splits the problem into two different projects.
No crawler arrived: you have an access problem. A robots rule, a firewall, a bot-mitigation product doing its job too well. Nothing you write will fix it.
Crawlers arrived, fetched cleanly, and you are still not a cited source: the problem is the content or its structure. Slower fix, different conversation.
Pull one week of logs and answer four questions
Ask whoever runs your hosting for seven days of access logs. Most platforms export them from a dashboard.
- Which AI crawlers reached you at all? Filter by user agent and list them. An absent crawler is the loudest result you can get.
- What status codes did they receive? Anything other than 200 or a sensible redirect needs explaining.
- Which URLs did they fetch, and which did they never touch? Compare against your commercially important pages.
- Does the HTML they were served hold your content? Fetch the same URL with JavaScript disabled and read what comes back.
Four answers, one afternoon, nothing bought. Clean access and complete HTML means you can stop blaming the plumbing and look at the writing.
Frequently asked questions
-
Is log file analysis the same as checking analytics?
No, and they capture different moments. Analytics records human sessions that ran your tracking script. A server log records every request that reached the server, bots and failed requests included. For crawler questions the log is the only reliable source, because bots do not run tracking scripts.
-
Do I need a tool to read log files?
Not for the first pass. Command-line tools handle a week of logs comfortably and a spreadsheet works for a smaller site. Log analyzers earn their place when volume is large or you want crawler trends across months.
-
Can I trust the user agent to identify a crawler?
Mostly, not entirely. User agents are self-reported and can be spoofed, so a claimed GPTBot may not be one. The major providers publish IP ranges you can verify against, worth doing before you make an access decision on what a header said.
-
Is this worth doing if my site is small?
Once, yes, then rarely. The first read either finds a blocking problem or rules one out, and both outcomes save months of guessing. After that a small site with stable infrastructure has little reason to look more than once or twice a year.