How AI Crawlers Like GPTBot and ClaudeBot Work
Updated 2026-07-23
How AI crawlers work
How AI crawlers work starts with a simple definition: an AI crawler is an automated program that fetches pages from the web on behalf of an AI company, either to build a search index used for retrieval, to gather material for training a model, or both. Functionally, an AI crawler works much like a traditional search engine crawler: it requests a page, reads the response, follows links it finds, and moves on to the next page, repeating that process across a large number of sites. Each AI crawler identifies itself with a distinct user agent string in its request headers, which is how a server operator can tell which crawler made a given request rather than treating all automated traffic as one undifferentiated category.
The documented crawler user agents to know
Several AI crawlers have publicly documented user agent names that a server operator can look for directly in access logs. GPTBot is the documented crawler associated with OpenAI. ClaudeBot is the documented crawler associated with Anthropic. PerplexityBot is the documented crawler associated with Perplexity. Googlebot and Bingbot, the crawlers behind Google Search and Bing respectively, also feed the retrieval systems behind Google AI Overviews and Bing Copilot even though they are not AI-specific crawlers by origin. Confirming which of these has requested a given page is a matter of searching server logs for the exact user agent string, rather than inferring crawler activity from indirect signals like traffic patterns alone.
What robots.txt does and does not control
robots.txt is the file that governs whether a given crawler, including an AI crawler, is allowed to access parts of a site, and most documented AI crawlers respect the rules a site publishes there for their specific user agent. This makes robots.txt the primary lever a publisher has for allowing or blocking AI crawler access at the access level. What robots.txt does not control is what happens with content the crawler was already allowed to fetch previously, or what an AI system does with content it accesses through means other than its own crawler, such as a user pasting a page's text directly into a chat. Blocking a crawler in robots.txt also does not retroactively remove anything the crawler already collected before the rule was added.
What you can observe versus what is inference
What a publisher can observe directly is server log data: which documented crawler user agents have requested which pages, and how often, over a given period. This is a genuine, first-party observation, not an inference. What is not observable from logs alone is what an AI company's system does with a page after fetching it, whether it was indexed for retrieval, used for training, or discarded. A crawl confirms access, not outcome. Assuming a crawl automatically means a page will show up as a citation, or that a lack of crawling explains every absence, both go beyond what a log entry actually proves, and the honest reading stays limited to what the log shows: this crawler requested this page at this time.
How to confirm crawler activity in your own logs
Confirming crawler activity starts with searching a site's raw access logs for the documented user agent strings covered above, GPTBot, ClaudeBot, PerplexityBot, Googlebot, and Bingbot, rather than relying on analytics tools that may filter automated traffic out by default. Most hosting platforms and content delivery networks provide some form of raw or filtered log access, and searching those logs for each user agent string shows whether and how often a given crawler has reached a page. llms.txt is a separate, related mechanism worth understanding alongside crawler activity, since it addresses what a site makes available to AI systems rather than which crawler accessed it.
Frequently asked questions
Do I need to allow every AI crawler for my page to be cited?
Not necessarily, but blocking a crawler in robots.txt generally prevents that specific system from fetching the page at all, which removes the page from consideration by that crawler's associated retrieval or training process.
Does a crawl from GPTBot or ClaudeBot guarantee my page gets cited?
No. A crawl confirms access to the page, not what happens afterward. Whether the content gets indexed, used, or cited is a separate step that is not directly observable from a log entry.
Where do I look to confirm AI crawler activity?
Search raw server access logs for the documented user agent strings, including GPTBot, ClaudeBot, PerplexityBot, Googlebot, and Bingbot, since standard analytics tools sometimes filter automated traffic out by default.