The short answer
AI crawlers are the bots that AI companies run to fetch web pages for model training, for a search index, or live when a user asks. GPTBot and ClaudeBot train, OAI-SearchBot and PerplexityBot index, and ChatGPT-User and Perplexity-User fetch on request. Google-Extended is a control token, not a crawler.
Updated Oct 18 2026
Why AI Crawlers matters
Blocking the wrong bot can remove you from answers, and allowing the wrong one can feed training you did not intend. The bots serve different purposes. The training and search bots follow robots.txt, but fetchers that a user triggers, such as ChatGPT-User and Perplexity-User, may not.
How to measure or use it
Read robots.txt and your server logs: which bots visit, which are blocked and whether the blocks are deliberate.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /How I handle it
I decide per bot, on purpose. I usually allow the search and live fetch bots that can send visibility, and decide separately on the training bots.
Related terms
- AI Crawl PathwayThe route AI bots can follow to discover, render, understand and revisit your important pages.Read
- llms.txtA proposed Markdown file, usually at the site root, that points language models to your key content.Read
- RetrievalHow an AI system finds candidate pages and passages before it writes an answer.Read
AI Crawlers questions
Does blocking GPTBot remove me from ChatGPT search?
Not by itself. OpenAI documents separate bots for training and for search, so they can be set differently.
How do I see which bots visit me?
Look at your server or CDN logs, or run my free AI crawler check to see how your robots rules treat each bot.
Go further