Definition
Crawlability is whether an automated client can reach a page on your site, fetch it, and receive readable content in the response. Indexability is the next question along, whether that fetched page is then eligible to be stored and returned. A page that fails crawlability cannot be ranked, cited or quoted anywhere.
At a glance
- CategoriesTechnical
- Related fieldsSEO, Technical
- DifficultyIntermediate
At a glance
- Crawlability covers four things in sequence: a discoverable URL, a permitted request, a good response and readable HTML.
- robots.txt decides which crawlers may request which paths, named agent by named agent.
- A sitemap helps discovery. It grants no access and guarantees no fetch.
- A 200 response proves the page was served, not that your content was in it.
- Crawlability failures are invisible in analytics, because a refused bot runs no tracking script.
The chain a bot walks before your page exists
Four things happen in order, and a break anywhere ends the sequence.
Discovery comes first. A crawler finds a URL through an internal link, an external link or your sitemap. The sitemaps protocol is the formal way to hand over a list, and an orphan page with no inbound link and no sitemap entry may never be found.
Permission comes second. The crawler checks robots.txt for a rule matching its user agent. RFC 9309 standardized this behavior, and the major crawlers honour it, AI ones included.
Response comes third. Your server returns a status code, and anything other than a 200 or a clean redirect stops the crawl there.
Content comes last. The HTML in that response either holds your words or holds an empty shell waiting on JavaScript. Not every crawler runs JavaScript, and the ones that do not see only what the server sent.
One stale rule can remove you from a whole category
Crawlability fails quietly. Nothing errors, nobody gets paged, the dashboards look normal, because a refused bot never ran your tracking script.
The shape it usually takes is this. A Disallow: /products/ goes into robots.txt during a replatform, to keep staging URLs out of the index. The replatform ships. The rule stays. Months later a shopper asks ChatGPT which brands make a particular thing, and every competitor gets named because their product pages were readable and yours were not.
The same ending arrives by other routes: a security product challenging an unfamiliar user agent, a framework rendering everything in the browser, a CDN serving an interstitial to any client without a browser fingerprint.
None of these are content problems. You can publish the best page in the category and it changes nothing, because the system choosing what to cite never saw it.
Run a crawl check without buying anything
An afternoon, using what you already have.
- Fetch your key pages with JavaScript disabled and read what comes back. Whatever is missing is also missing for an AI crawler.
- Read robots.txt line by line. Check the wildcard block first, then every named agent. Anything nobody can explain, treat as suspect.
- Confirm the sitemap matches reality. No 404s, no redirect chains, no pages you quietly retired last quarter.
- Count crawler hits by section in a week of server logs. A section no bot has touched is either unreachable or unlinked.
- Compare crawled against indexed. Search Console names pages Google fetched and declined to keep, with reasons specific enough to act on.
All five cost engineering time and nothing else, and together they decide whether your content freshness work reaches a model at all.
Where crawlability sits in the chain
| Stage | The question it answers | What usually breaks it |
|---|---|---|
| Discoverable | Can a bot find the URL? | Orphan pages, missing sitemap entries |
| Permitted | Is this agent allowed to request it? | A stale robots.txt Disallow |
| Fetchable | Does the server return the page? | Timeouts, 403s from bot mitigation |
| Readable | Does the HTML hold the content? | Client-side rendering |
| Indexable | Is the page eligible to be stored? | noindex tags, canonical conflicts |
| Extractable | Can a clean answer be lifted from it? | Walls of text, content locked in images |
Frequently asked questions
-
Is crawlability the same as indexability?
No, and the order matters. Crawlability asks whether a bot can reach and read the page. Indexability asks whether the page it read is eligible to be stored and served. Every indexable page is crawlable first. Plenty of crawlable pages are kept out of the index on purpose.
-
How do I know whether AI crawlers can read my site?
Two checks answer it. Load a key page with JavaScript disabled and see whether your content is in the HTML. Then search a week of server logs for known AI user agents and confirm they arrived and got a 200. Between them they find most failures.
-
Does having a sitemap make my pages crawlable?
No. A sitemap helps a crawler discover URLs it would otherwise miss, which is discovery rather than permission. If robots.txt blocks the path or the server refuses the request, listing the URL changes nothing. Both have to be right.
-
Will fixing crawlability get my brand cited in AI answers?
It will not, on its own, and it is worth being straight about that. Crawlability removes the reason you cannot be cited. Whether you then get cited depends on whether the page says something clear, specific and worth quoting. Fixing access is necessary and nowhere near sufficient.