IABIGFR · Français

AI crawlers · Updated 2026-11-06 · Markdown version

How to opt out of AI training - OpenAI, Google, Apple, Anthropic, Meta and more

The robots.txt tokens and settings that opt your site out of AI training at each major company, what each opt-out covers, and what it leaves untouched.

There is no single switch that keeps your website out of every AI training dataset. Each company documents its own opt-out, usually a robots.txt token, sometimes a meta tag, and each one covers something slightly different. Some tokens are real crawlers, others are only a preference attached to an existing search crawler. None of the companies promises to remove what it has already collected. This guide lists the training opt-out for each major company as documented in November 2026, explains what it does and does not cover, gives a complete robots.txt and points to the platform settings that write it for you. For the robots.txt syntax itself and the RFC 9309 mistakes to avoid, see block AI training with robots.txt.

The opt-out for each company

Company What to disallow Crawls itself? What the opt-out covers, per the company
OpenAI GPTBot Yes Content should not be used to train OpenAI's generative AI foundation models
Google Google-Extended No, preference token Training future Gemini models for Gemini Apps and the Vertex AI API, and grounding in Gemini Apps and Vertex AI
Apple Applebot-Extended No, preference token Training Apple's foundation models (Apple Intelligence, Services, Developer Tools)
Anthropic ClaudeBot Yes Future content excluded from training datasets
Meta meta-externalagent Yes Crawling for uses such as training foundation models or improving products by indexing content
Common Crawl CCBot Yes New pages kept out of future open web archives
Amazon Amazonbot, or the noarchive meta tag Yes Amazonbot data may be used to train Amazon AI models; noarchive means do not use the page for model training
Mistral AI MistralAI-Training Yes Crawling to build datasets for training Mistral models
Microsoft No token; noarchive or nocache meta tags Bingbot noarchive excludes content from training Microsoft's foundation models; nocache limits it to URL, title and snippet
ByteDance Bytespider Yes Not officially documented

Each token has a page in our AI crawler directory with its user-agent string, IP list and verification method where one exists.

A few entries need more explanation.

Google and Apple use preference tokens. Google-Extended and Applebot-Extended never appear in your logs. Google crawls with its usual user agents and Apple with Applebot; the extra token only tells each company how it may use what was crawled. Blocking Googlebot or Applebot to refuse training would also take you out of Google Search or out of Spotlight, Siri and Safari. Google-Extended has a further consequence: according to Google, it also covers grounding in Gemini Apps and in Grounding with Google Search on Vertex AI, so you cannot stay a Gemini source while refusing training with this token alone.

Meta's crawler is broader than training. Meta describes meta-externalagent as crawling "for use cases such as training foundation AI models or improving products by indexing content directly". Meta runs separate crawlers for Meta AI search (meta-webindexer), advertising (meta-externalads) and user-requested fetches (meta-externalfetcher). Meta also says it relies on robots.txt "rather than non-standard formats like NoAI tags", so a noai meta tag is not a documented opt-out at Meta.

Common Crawl is not an AI company, but its archive is free to download and widely reused, including by third parties to train models. Its CCBot page does not mention AI training. Blocking CCBot is how you keep new pages out of future archives. Common Crawl also publishes an Opt-Out Ledger listing legal opt-out requests it has received; requests go to info@commoncrawl.org. Neither page says whether pages already archived are removed.

Amazon has two options. Disallowing Amazonbot stops a crawler whose data serves Amazon's products and services as well as training. The page-level noarchive meta tag, which Amazon documents as "do not use the page for model training", is a narrower alternative. Amazon's search agent, Amzn-SearchBot, is not used for generative AI training.

Microsoft has no robots.txt token for AI. Bing's guidelines tie Copilot and training to meta tags instead. noarchive keeps content out of Copilot answers and grounding results as well as training, so it is not a training-only switch.

Bytespider is unverified. We found no official ByteDance documentation describing the crawler, its use of data or its robots.txt handling. A Disallow rule records your refusal, but nothing confirms it is honored.

Crawlers that are not for training

Some agents often appear on AI blocklists even though their operators say they are not used for training. Blocking them costs visibility without helping your opt-out:

  • OAI-SearchBot: ChatGPT search. Blocking it removes you from ChatGPT search answers.
  • Claude-SearchBot and Claude-User: Claude's search index and live fetches.
  • PerplexityBot: Perplexity says it is not used to crawl content for AI foundation models.
  • DuckAssistBot: DuckDuckGo says the data is not used in any way to train AI models.
  • Amzn-SearchBot and MistralAI-Index: search indexing, not used for training according to Amazon and Mistral.

What an opt-out does not cover

Content already collected. None of the official pages we checked promises to delete or stop using data gathered before you added a rule. Anthropic explicitly speaks of excluding "future" content; OpenAI, Google, Apple and Common Crawl say nothing on the subject. Squarespace's help center says the same about its own blocking setting: it does not retroactively remove content that was already scraped.

Search and AI answers. Training and search are separate controls almost everywhere. Blocking GPTBot does not remove you from ChatGPT search, and blocking ClaudeBot does not remove you from Claude's answers. Google's AI Overviews and AI Mode are not covered by Google-Extended: they are part of Google Search, and Google points to Googlebot rules in robots.txt and to the nosnippet, data-nosnippet, max-snippet and noindex controls to limit how your content is used there. Apple's AI answers in Siri and Search are controlled with nosnippet, not Applebot-Extended: Apple says it will not use nosnippet content as context for those answers, while the page stays discoverable in Spotlight, Siri and Safari.

User-initiated fetchers. OpenAI says robots.txt may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it, and Meta says meta-externalfetcher may bypass it. These agents fetch pages to answer a user, not to build training sets, but their operators do not promise to follow your file.

Copies held by others. If your content sits in an archive, a dataset or another site, opting out of one crawler does not reach those copies.

Impersonators. robots.txt is a request. A scraper that spoofs a known user agent will not follow it. Only filtering by IP address, against the ranges each company publishes, stops those, and Meta, Bytespider and the preference tokens have no published ranges to check.

Subdomains. Each host serves its own robots.txt. Anthropic explicitly tells site owners to add the rule to every subdomain, and the same logic applies to every crawler.

Company-neutral signals are still drafts

Listing tokens one company at a time does not scale, and several proposals try to express a single "no AI training" preference. None is a finished standard yet, so treat them as a complement to the per-company tokens, not a replacement:

  • Content Signals (Content-signal: search=yes, ai-train=no in robots.txt) is a Cloudflare proposal that its managed robots.txt adds automatically. Cloudflare itself notes that it only expresses a preference.
  • The IETF AI Preferences (aipref) working group is drafting a shared vocabulary for such preferences. Anthropic says it is rolling out support for one individual Internet-Draft, draft-kelley-aipref-training-preference-signal, which uses a reserved robots.txt group (User-Agent: __usage_ai-training followed by Disallow: /). Anthropic adds that a group naming ClaudeBot takes priority over that signal.

No other company in the table above documents support for these signals as of this writing, so keep the explicit tokens.

A complete robots.txt

This file refuses training at every company listed above while keeping search engines and AI search crawlers allowed. Under RFC 9309, a crawler that finds a group naming it ignores the * group, so the exclusions of the * group are repeated in the named search group.

# robots.txt - https://www.example.com/robots.txt
# Goal: opt out of AI training, stay visible in search and AI answers.

# 1. AI training: refused
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: Bytespider
Disallow: /

# 2. AI search and user-initiated fetches: allowed
#    Named groups ignore the * group, so repeat its exclusions here.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-webindexer
User-agent: Amzn-SearchBot
User-agent: MistralAI-Index
User-agent: MistralAI-User
Disallow: /admin/
Allow: /

# 3. Everyone else, including Googlebot, Bingbot and Applebot
User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Adjust it to your goal:

  • To stay a source for Gemini answers, remove Google-Extended from the first group and accept that your content may also be used for Gemini training.
  • To keep Amazon's products using your pages while refusing training, remove Amazonbot and add noarchive to your pages instead.
  • For Microsoft, robots.txt is not the lever. If you want to refuse training by Microsoft's models, add the meta tag below, knowing it also keeps the page out of Copilot answers.
<meta name="robots" content="noarchive">

A generic robots meta tag is read by every crawler that supports it, so check what noarchive means for the other engines you care about before applying it site-wide.

The file is explained line by line in block AI training with robots.txt, along with the common RFC 9309 mistakes. To build a variant for your own site, use the robots.txt generator.

Platform settings

If you cannot edit robots.txt directly, your host or CDN may write the rules for you, with its own list of bots.

  • Cloudflare classifies AI bots into Search, Agent and Training categories and lets you block each category at the edge, which is enforcement rather than a request. Cloudflare announced the deprecation of the older single Block AI bots setting on September 15, 2026. Details in Cloudflare and AI crawlers.
  • Squarespace does not let you edit robots.txt. Its Block known artificial intelligence crawlers checkbox, under Settings > Crawlers, adds a Squarespace-maintained list to robots.txt that mixes training crawlers with agents that are not documented as training crawlers, such as DuckAssistBot, as well as obsolete tokens such as anthropic-ai. See Squarespace and AI crawlers.
  • WordPress, Wix, Shopify and Webflow each handle robots.txt differently: see the guides for WordPress, Wix, Shopify and Webflow.

Whatever the platform, open https://your-domain/robots.txt after changing a setting and check that the expected tokens are there and that the file returns HTTP 200. To see which training tokens other sites actually disallow, the weekly AI crawler census tracks the robots.txt files of the top 10,000 sites.

Official sources

Free tools

Related guides