Jahanzaib

Free tool

robots.txt generator for AI crawlers

I built this to write a robots.txt that treats AI crawlers by what they do: training, search and retrieval, or a fetch a person asked for. Pick a preset, change any bot, add your general rules, and test the file before you upload it.

To see how your live robots.txt treats these bots, run my AI crawler check. For what each kind of bot is for, read my glossary entry on AI crawlers, and for Meta's terms, my post on the robots.txt lines Meta reads.

Start from a preset

A preset sets the bot rules only. Your general rules below stay as they are.

Training crawlers

Collect pages to train models. Blocking most of them does not remove you from AI search answers. Google-Extended is the exception: it also controls whether Gemini apps use your pages to ground their answers.

  • GPTBotOpenAI · LLM training data

    OpenAI says disallowing it tells OpenAI your content should not be used to train its foundation models. platform.openai.com

    Rule for GPTBot

  • ClaudeBotAnthropic · Claude model training

    Anthropic says blocking it signals that your future content should be left out of its training datasets. support.claude.com

    Rule for ClaudeBot

  • Google-ExtendedGoogle · Gemini/Vertex AI training

    A control token, not a separate crawler. Google says it governs Gemini training and also grounding in Gemini Apps and Grounding with Google Search on Vertex AI, so blocking it can keep your pages out of those Gemini answers. Google says it does not affect inclusion or ranking in Google Search. developers.google.com

    Rule for Google-Extended

  • BytespiderByteDance · Doubao LLM training

    I could not find ByteDance documentation for this token, so I cannot confirm ByteDance reads it.

    Rule for Bytespider

  • CCBotCommon Crawl · Open ML training dataset

    Common Crawl documents a CCBot block in robots.txt and warns that some crawlers falsely identify themselves as CCBot. commoncrawl.org

    Rule for CCBot

  • Meta-ExternalAgentMeta · Meta AI model training

    Meta says it crawls for uses such as training foundation models and indexing, and documents robots.txt groups for it. developers.facebook.com

    Rule for Meta-ExternalAgent

  • AmazonbotAmazon · Amazon products and AI model training

    Amazon says it respects robots.txt user agent and allow or disallow rules, and does not support Crawl-delay. developer.amazon.com

    Rule for Amazonbot

  • MistralAI-TrainingMistral · Mistral model training

    Mistral says webmasters can disallow it in robots.txt, and that it is not used for search or live user queries. docs.mistral.ai

    Rule for MistralAI-Training

  • Applebot-ExtendedApple · Apple foundation model training

    Apple says it does not crawl pages, only decides how Applebot data is used, and that pages which disallow it can still appear in Apple search results. support.apple.com

    Rule for Applebot-Extended

Search and retrieval

Build the indexes AI assistants search when they answer. Blocking them can keep you out of those answers.

  • OAI-SearchBotOpenAI · ChatGPT search indexing

    OpenAI says sites that block it are not shown in ChatGPT search answers, though they can still appear as navigational links. platform.openai.com

    Rule for OAI-SearchBot

  • PerplexityBotPerplexity · Perplexity search indexing

    Perplexity says it surfaces and links sites in its search results and is not used to crawl content for foundation models. docs.perplexity.ai

    Rule for PerplexityBot

  • Claude-SearchBotAnthropic · Claude search indexing

    Anthropic says blocking it stops Claude from indexing your content for search, and that its bots honor robots.txt. support.claude.com

    Rule for Claude-SearchBot

  • GrokxAI · Grok AI search

    I could not find xAI documentation for this token, so I cannot confirm xAI reads it.

    Rule for Grok

  • BravebotBrave · Brave Search AI indexing

    Brave says its search crawler does not use a separate user agent and will not crawl what your Googlebot rules block, so a Bravebot group may have no effect. search.brave.com

    Rule for Bravebot

  • Meta-WebIndexerMeta · Meta AI search results

    Meta says allowing it in robots.txt helps Meta AI cite and link to your content. developers.facebook.com

    Rule for Meta-WebIndexer

  • Amzn-SearchBotAmazon · Amazon search experiences such as Alexa

    Amazon says it respects robots.txt, and that when your file does not name it but allows other search bots, it follows the rules you give those bots. developer.amazon.com

    Rule for Amzn-SearchBot

  • MistralAI-IndexMistral · Mistral search indexing

    Mistral lists it as its search indexer, separate from MistralAI-User and MistralAI-Training, and says it is for indexing only. docs.mistral.ai

    Rule for MistralAI-Index

User triggered fetchers

Open a page when a person asks an assistant about it. Some vendors say these may not follow robots.txt.

  • ChatGPT-UserOpenAI · Real time user queries

    OpenAI says it is not used to crawl the web automatically and that, because a user starts these fetches, robots.txt rules may not apply. platform.openai.com

    Rule for ChatGPT-User

  • Claude-UserAnthropic · Claude user queries

    Anthropic says blocking it stops Claude from fetching your pages in answer to a user, and that its bots honor robots.txt. support.claude.com

    Rule for Claude-User

  • Perplexity-UserPerplexity · Perplexity user queries

    Perplexity says that since a user requested the fetch, this fetcher generally ignores robots.txt rules. docs.perplexity.ai

    Rule for Perplexity-User

  • Meta-ExternalFetcherMeta · Meta AI user responses

    Meta says it may bypass robots.txt because it performs fetches a user asked for. developers.facebook.com

    Rule for Meta-ExternalFetcher

  • MistralAI-UserMistral · Le Chat user queries

    Mistral says it governs which sites user requests in its assistant can reach, and that it does not crawl automatically or for training. docs.mistral.ai

    Rule for MistralAI-User

  • Amzn-UserAmazon · Live answers for Alexa users

    Amazon says that because a user can start these fetches, it may not follow all robots.txt directives. developer.amazon.com

    Rule for Amzn-User

  • DuckAssistBotDuckDuckGo · DuckDuckGo AI assisted answers

    DuckDuckGo says disallowing it in robots.txt stops it crawling your site once the change takes effect after 72 hours, and does not affect your DuckDuckGo search rankings. duckduckgo.com

    Rule for DuckAssistBot

No rule means the bot gets no group of its own and follows the Every other crawler group.

General rules

Every other crawler (User-agent: *)

Written into the wildcard group and into every bot you allow, because a bot with its own group ignores the wildcard group.

Use the full address with http or https. A relative path is refused.

Google ignores Crawl-delay. Bing and Anthropic document support for it. To slow Googlebot, use your server or CDN.

The product token only, as the vendor documents it. Each one is blocked from the whole site.

Your robots.txt

Upload it to the root of your site, so it opens at /robots.txt on the same host and protocol. Each subdomain needs its own file.

Who can fetch what

The file above, parsed with the robots-parser package. Nothing is fetched: this tests the text, not your live site.

Access for each user agent at / and at /blog/
User agent//blog/
GPTBotOpenAIBlockedline 14Blockedline 14
ClaudeBotAnthropicBlockedline 14Blockedline 14
Google-ExtendedGoogleBlockedline 14Blockedline 14
BytespiderByteDanceBlockedline 14Blockedline 14
CCBotCommon CrawlBlockedline 14Blockedline 14
Meta-ExternalAgentMetaBlockedline 14Blockedline 14
AmazonbotAmazonBlockedline 14Blockedline 14
MistralAI-TrainingMistralBlockedline 14Blockedline 14
Applebot-ExtendedAppleBlockedline 14Blockedline 14
OAI-SearchBotOpenAIAllowedline 25Allowedline 25
PerplexityBotPerplexityAllowedline 25Allowedline 25
Claude-SearchBotAnthropicAllowedline 25Allowedline 25
GrokxAIAllowedline 25Allowedline 25
BravebotBraveAllowedline 25Allowedline 25
Meta-WebIndexerMetaAllowedline 25Allowedline 25
Amzn-SearchBotAmazonAllowedline 25Allowedline 25
MistralAI-IndexMistralAllowedline 25Allowedline 25
ChatGPT-UserOpenAIAllowedline 35Allowedline 35
Claude-UserAnthropicAllowedline 35Allowedline 35
Perplexity-UserPerplexityAllowedline 35Allowedline 35
Meta-ExternalFetcherMetaAllowedline 35Allowedline 35
MistralAI-UserMistralAllowedline 35Allowedline 35
Amzn-UserAmazonAllowedline 35Allowedline 35
DuckAssistBotDuckDuckGoAllowedline 35Allowedline 35
GooglebotGooglebot (Google Search)Allowedline 39Allowedline 39
BingbotBingbot (Bing Search)Allowedline 39Allowedline 39

robots.txt is a request, not a lock

Well behaved crawlers read it and follow it. It does not stop a bot that chooses to ignore it, and it does not hide a page from people. For private content, use a login or block the bot at your server or CDN. These vendors say their user triggered fetchers may not follow robots.txt:

Coverage

What it writes

  • AI crawlers grouped by job

    Training crawlers, search and retrieval crawlers, and user triggered fetchers, each with a line from the vendor’s own documentation and a link to it.

  • Three presets

    Allow all, allow search but block training, or block all AI. A preset changes the bot rules and keeps your paths and sitemaps.

  • Your general rules in every allowed group

    A bot with its own group ignores User-agent: *, so the paths you disallow are copied into each allowed group as well as the wildcard group.

  • Sitemap lines as full URLs

    Each Sitemap line must be an absolute http or https address. A relative path or anything else is refused and shown as an error.

  • Crawl delay, honestly labelled

    An optional Crawl-delay with a comment in the file that Google ignores it. Bing and Anthropic document support for it.

  • A validator on the output

    The generated text is parsed with robots-parser to show, per user agent, whether / and a sample path can be fetched and which line decided it.

Method

How it works

  1. 1

    Pick a preset

    Start from allow all, allow search but block training, or block all AI. If you want AI search visibility without feeding training, start from the middle one.

  2. 2

    Adjust each bot

    Set any bot to Allow, Block or No rule. No rule leaves it out, so it follows whatever you set for every other crawler.

  3. 3

    Add your general rules

    Paths to keep crawlers out of, your sitemap URLs, other user agents to block, and an optional crawl delay. Lines that fail a check are listed and left out.

  4. 4

    Test, then upload

    Read the validator table, then copy or download the file and put it at the root of your site. Check it again in your browser at /robots.txt.

Questions

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT search?

Not by itself. OpenAI documents separate bots: GPTBot collects content that may train its models, OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User handles fetches a user starts. The allow search, block training preset sets them that way.

Does blocking Google-Extended hurt my Google rankings?

Google says it does not. Google-Extended is a control token, not a separate crawler. It decides whether content Google already crawls may be used to train Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion or ranking in Google Search.

Will robots.txt stop every AI bot?

No. robots.txt is a request that well behaved crawlers choose to follow. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Meta says Meta-ExternalFetcher may bypass them, because a person asked for the page. To enforce a block, use a login or a rule at your server or CDN.

Why are my disallowed paths repeated in several groups?

A crawler follows only the most specific group that names it and ignores the User-agent: * group. If I named OAI-SearchBot with Allow: / and left /admin/ only in the wildcard group, OAI-SearchBot could read /admin/. So the tool writes your paths into every group of bots you allow.

Why is the User-agent: * group at the bottom?

Under the robots.txt standard the order of groups does not matter, because a crawler picks the most specific match. Some simple parsers stop at the first group that matches, so I put the named groups first and the wildcard group last. Sitemap lines go at the end because they belong to no group.

Does Google read Crawl-delay?

No. Google says it supports only the user-agent, allow, disallow and sitemap fields, and other fields such as crawl-delay are not supported. Bing has documented support for it, and Anthropic says it supports it. The tool adds a comment saying so whenever you set one.

Does the tool send my rules anywhere?

No. The file is built and tested in your browser, and nothing you type leaves the page. The validator parses the generated text with the robots-parser package. It does not fetch your site.

Where do I put the file?

At the root of the host it covers, so it opens at /robots.txt on that exact host and protocol. A subdomain needs its own file. Crawlers cache robots.txt, so a change can take a while to be picked up; OpenAI and Meta each mention about 24 hours.

More free tools

Related tools

See every free tool

Rather hand it over?

I can find the fixes and ship them for you

I run SEO and answer engine optimization as one managed program: the pages, the technical fixes, the markup and the mentions that AI assistants read. You get a plain monthly report on what changed.

See how I run AEO