Skip to main content
smnify.

AI Crawler robots.txt Builder

Pick Allow, Block or No rule for each AI crawler and copy the matching robots.txt groups. The list separates bots that collect training data from bots that power AI search and bots that fetch a page because a user asked. robots.txt is advisory: it only works for crawlers that choose to follow it, and some user-triggered fetchers say they may not.

Runs in your browser

Free, no sign-up. The file is built in your browser. Nothing is uploaded, stored or sent to analytics, and your live site is never fetched.

Choose a rule per crawler

Leave a crawler on "No rule" to say nothing about it. Tokens last verified against each operator's documentation on 2026-10-12.

  • GPTBot

    OpenAI - Training / data collection

    Rule for GPTBot

    Crawls content that may be used to train OpenAI generative models. Source

  • OAI-SearchBot

    OpenAI - Search indexing

    Rule for OAI-SearchBot

    Surfaces sites in ChatGPT search features. Blocking it keeps you out of ChatGPT search answers. Source

  • ChatGPT-User

    OpenAI - User-requested fetch

    Rule for ChatGPT-User

    Visits pages when a user asks ChatGPT or a custom GPT. OpenAI says robots.txt rules may not apply to these user-initiated visits. Source

  • ClaudeBot

    Anthropic - Training / data collection

    Rule for ClaudeBot

    Collects web content that could contribute to training Anthropic models. Source

  • Claude-SearchBot

    Anthropic - Search indexing

    Rule for Claude-SearchBot

    Crawls to improve the quality and relevance of Claude search results. Source

  • Claude-User

    Anthropic - User-requested fetch

    Rule for Claude-User

    Accesses sites when a Claude user asks a question that needs it. Source

  • Google-Extended

    Google - Training / data collection

    Rule for Google-Extended

    robots.txt-only token (no separate user agent). Controls use of crawled content for Gemini training and grounding. Does not affect Google Search. Source

  • PerplexityBot

    Perplexity - Search indexing

    Rule for PerplexityBot

    Surfaces and links sites in Perplexity search results. Perplexity says it is not used to crawl for AI foundation models. Source

  • Perplexity-User

    Perplexity - User-requested fetch

    Rule for Perplexity-User

    Visits a page when a user asks a question. Perplexity says it generally ignores robots.txt because a user triggered it. Source

  • Applebot-Extended

    Apple - Training / data collection

    Rule for Applebot-Extended

    Opt-out token for training Apple's foundation models. It does not crawl; Applebot does the crawling and stays allowed for search. Source

  • CCBot

    Common Crawl - Training / data collection

    Rule for CCBot

    Builds the open Common Crawl dataset, which many third parties use for AI training. Source

  • Meta-ExternalAgent

    Meta - Mixed / general

    Rule for Meta-ExternalAgent

    Crawls for training foundation models and for indexing content directly. Matching is case-insensitive; Meta documents the token in lowercase. Source

  • Amazonbot

    Amazon - Mixed / general

    Rule for Amazonbot

    Improves Amazon products and services and may be used to train Amazon AI models. Honours robots.txt. Source

  • Bytespider

    ByteDance - Training / data collection

    Rule for Bytespider

    ByteDance crawler. ByteDance publishes no official robots.txt documentation we could verify, and third parties report it may not honour robots.txt. Treat as unverified.

Other settings

robots.txt

Merge these groups into your existing robots.txt; do not replace rules for Googlebot or Bingbot by accident. robots.txt is advisory: well-behaved crawlers follow it, others may not.

Need help implementing these improvements?

We fix the issues these checks surface: metadata, structured data, performance and security headers, shipped and verified on your stack.

Get help with your site

How to use the AI Crawler robots.txt Builder

  1. For each crawler, choose Allow, Block or No rule. No rule writes nothing for that bot.
  2. Use the shortcuts to block training and data-collection bots or allow search bots, then adjust individual crawlers.
  3. Choose what applies to every other crawler (User-agent: *): allow all, block all, or no rule.
  4. Optionally enter your sitemap URL; it is added as a Sitemap line.
  5. Press Copy or Download and merge the groups into your existing robots.txt at the root of your site.

Worked example

Sample data

Input
GPTBot: Block
ClaudeBot: Block
OAI-SearchBot: Allow
All other crawlers: Allow
Sitemap: https://a.example/sitemap.xml
Output
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://a.example/sitemap.xml

Crawlers with the same rule share one group. This blocks two training crawlers while keeping ChatGPT search eligible.

How it works

Training, search and user-fetch bots

Training or data-collection bots (for example GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) gather content that may be used to train models. Search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index pages so they can appear in AI search answers. User-fetch bots (ChatGPT-User, Claude-User, Perplexity-User) visit a page because a person asked an assistant about it.

Blocking a training bot does not remove you from that vendor's search, and blocking a search bot can remove you from its answers. Choose per purpose rather than blocking everything by default.

robots.txt is advisory

robots.txt is a request, not an access control. Reputable crawlers follow it, others may not. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores robots.txt because a user triggered the visit. To truly restrict access use authentication or server-side rules. Changes can take about 24 hours to be picked up.

Tokens that are not simple crawlers

Google-Extended and Applebot-Extended are robots.txt-only controls: they have no separate crawler of their own. Google-Extended does not affect Google Search; Applebot-Extended does not stop Applebot from crawling for search.

Bytespider (ByteDance) has no official documentation we could verify, and third parties report it may not honour robots.txt. It is included but marked unverified.

Sources and last verified date

Last verified: 12 October 2026. Tokens change; re-check the operator pages before relying on them.

OpenAI: https://developers.openai.com/api/docs/bots. Anthropic: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler. Google: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers. Perplexity: https://docs.perplexity.ai/guides/bots. Apple: https://support.apple.com/en-us/119829. Meta: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/. Common Crawl: https://commoncrawl.org/ccbot. Amazon: https://developer.amazon.com/amazonbot.

Inputs, outputs and limits

Inputs

  • Allow / Block / No rule for 14 crawlers
  • Rule for all other crawlers
  • Optional sitemap URL

Outputs

  • robots.txt groups with exact user-agent tokens
  • Per-crawler purpose and source link
  • Copy and download as robots.txt

Limits

  • Only the listed crawlers; new bots appear often.
  • Rules apply to the whole site (Allow: / or Disallow: /), not to individual paths.
  • Cannot prove a crawler obeys the file or check your live robots.txt.
  • Output must be merged by hand with any existing robots.txt.

Processing

  • The file is built in your browser. Nothing is uploaded, stored or sent to analytics, and your live site is never fetched.

Frequently asked questions

Will blocking GPTBot remove me from ChatGPT search?

No. According to OpenAI, GPTBot relates to training and OAI-SearchBot controls ChatGPT search appearance. They are separate tokens.

Does blocking Google-Extended affect my Google rankings?

Google says Google-Extended does not affect Google Search inclusion or ranking. It controls use of content for Gemini training and grounding.

Is robots.txt enough to keep AI from using my content?

No. It is advisory. Compliant crawlers follow it, user-triggered fetchers may not, and content already collected is not removed. Use access controls for anything private.

Should I block everything?

It depends on your goals. Blocking search and user-fetch bots can reduce how often AI assistants show or cite your pages. Many sites block only training bots.

Can I block only part of my site?

Not in this tool, which writes whole-site rules. You can edit the output to add specific paths after Disallow or Allow.

Need help implementing these improvements?

We fix the issues these checks surface: metadata, structured data, performance and security headers, shipped and verified on your stack.

Get help with your siteWeb development