How to opt out of AI training - OpenAI, Google, Apple, Anthropic, Meta and more
The robots.txt tokens and settings that opt your site out of AI training at each major company, what each opt-out covers, and what it leaves untouched.
There is no single switch that keeps your website out of every AI training dataset. Each company documents its own opt-out, usually a robots.txt token, sometimes a meta tag, and each one covers something slightly different. Some tokens are real crawlers, others are only a preference attached to an existing search crawler. None of the companies promises to remove what it has already collected. This guide lists the training opt-out for each major company as documented in November 2026, explains what it does and does not cover, gives a complete robots.txt and points to the platform settings that write it for you. For the robots.txt syntax itself and the RFC 9309 mistakes to avoid, see block AI training with robots.txt.
The opt-out for each company
| Company | What to disallow | Crawls itself? | What the opt-out covers, per the company |
|---|---|---|---|
| OpenAI | GPTBot |
Yes | Content should not be used to train OpenAI's generative AI foundation models |
Google-Extended |
No, preference token | Training future Gemini models for Gemini Apps and the Vertex AI API, and grounding in Gemini Apps and Vertex AI | |
| Apple | Applebot-Extended |
No, preference token | Training Apple's foundation models (Apple Intelligence, Services, Developer Tools) |
| Anthropic | ClaudeBot |
Yes | Future content excluded from training datasets |
| Meta | meta-externalagent |
Yes | Crawling for uses such as training foundation models or improving products by indexing content |
| Common Crawl | CCBot |
Yes | New pages kept out of future open web archives |
| Amazon | Amazonbot, or the noarchive meta tag |
Yes | Amazonbot data may be used to train Amazon AI models; noarchive means do not use the page for model training |
| Mistral AI | MistralAI-Training |
Yes | Crawling to build datasets for training Mistral models |
| Microsoft | No token; noarchive or nocache meta tags |
Bingbot | noarchive excludes content from training Microsoft's foundation models; nocache limits it to URL, title and snippet |
| ByteDance | Bytespider |
Yes | Not officially documented |
Each token has a page in our AI crawler directory with its user-agent string, IP list and verification method where one exists.
A few entries need more explanation.
Google and Apple use preference tokens. Google-Extended and Applebot-Extended never appear in your logs. Google crawls with its usual user agents and Apple with Applebot; the extra token only tells each company how it may use what was crawled. Blocking Googlebot or Applebot to refuse training would also take you out of Google Search or out of Spotlight, Siri and Safari. Google-Extended has a further consequence: according to Google, it also covers grounding in Gemini Apps and in Grounding with Google Search on Vertex AI, so you cannot stay a Gemini source while refusing training with this token alone.
Meta's crawler is broader than training. Meta describes meta-externalagent as crawling "for use cases such as training foundation AI models or improving products by indexing content directly". Meta runs separate crawlers for Meta AI search (meta-webindexer), advertising (meta-externalads) and user-requested fetches (meta-externalfetcher). Meta also says it relies on robots.txt "rather than non-standard formats like NoAI tags", so a noai meta tag is not a documented opt-out at Meta.
Common Crawl is not an AI company, but its archive is free to download and widely reused, including by third parties to train models. Its CCBot page does not mention AI training. Blocking CCBot is how you keep new pages out of future archives. Common Crawl also publishes an Opt-Out Ledger listing legal opt-out requests it has received; requests go to info@commoncrawl.org. Neither page says whether pages already archived are removed.
Amazon has two options. Disallowing Amazonbot stops a crawler whose data serves Amazon's products and services as well as training. The page-level noarchive meta tag, which Amazon documents as "do not use the page for model training", is a narrower alternative. Amazon's search agent, Amzn-SearchBot, is not used for generative AI training.
Microsoft has no robots.txt token for AI. Bing's guidelines tie Copilot and training to meta tags instead. noarchive keeps content out of Copilot answers and grounding results as well as training, so it is not a training-only switch.
Bytespider is unverified. We found no official ByteDance documentation describing the crawler, its use of data or its robots.txt handling. A Disallow rule records your refusal, but nothing confirms it is honored.
Crawlers that are not for training
Some agents often appear on AI blocklists even though their operators say they are not used for training. Blocking them costs visibility without helping your opt-out:
OAI-SearchBot: ChatGPT search. Blocking it removes you from ChatGPT search answers.Claude-SearchBotandClaude-User: Claude's search index and live fetches.PerplexityBot: Perplexity says it is not used to crawl content for AI foundation models.DuckAssistBot: DuckDuckGo says the data is not used in any way to train AI models.Amzn-SearchBotandMistralAI-Index: search indexing, not used for training according to Amazon and Mistral.
What an opt-out does not cover
Content already collected. None of the official pages we checked promises to delete or stop using data gathered before you added a rule. Anthropic explicitly speaks of excluding "future" content; OpenAI, Google, Apple and Common Crawl say nothing on the subject. Squarespace's help center says the same about its own blocking setting: it does not retroactively remove content that was already scraped.
Search and AI answers. Training and search are separate controls almost everywhere. Blocking GPTBot does not remove you from ChatGPT search, and blocking ClaudeBot does not remove you from Claude's answers. Google's AI Overviews and AI Mode are not covered by Google-Extended: they are part of Google Search, and Google points to Googlebot rules in robots.txt and to the nosnippet, data-nosnippet, max-snippet and noindex controls to limit how your content is used there. Apple's AI answers in Siri and Search are controlled with nosnippet, not Applebot-Extended: Apple says it will not use nosnippet content as context for those answers, while the page stays discoverable in Spotlight, Siri and Safari.
User-initiated fetchers. OpenAI says robots.txt may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it, and Meta says meta-externalfetcher may bypass it. These agents fetch pages to answer a user, not to build training sets, but their operators do not promise to follow your file.
Copies held by others. If your content sits in an archive, a dataset or another site, opting out of one crawler does not reach those copies.
Impersonators. robots.txt is a request. A scraper that spoofs a known user agent will not follow it. Only filtering by IP address, against the ranges each company publishes, stops those, and Meta, Bytespider and the preference tokens have no published ranges to check.
Subdomains. Each host serves its own robots.txt. Anthropic explicitly tells site owners to add the rule to every subdomain, and the same logic applies to every crawler.
Company-neutral signals are still drafts
Listing tokens one company at a time does not scale, and several proposals try to express a single "no AI training" preference. None is a finished standard yet, so treat them as a complement to the per-company tokens, not a replacement:
- Content Signals (
Content-signal: search=yes, ai-train=noin robots.txt) is a Cloudflare proposal that its managed robots.txt adds automatically. Cloudflare itself notes that it only expresses a preference. - The IETF AI Preferences (aipref) working group is drafting a shared vocabulary for such preferences. Anthropic says it is rolling out support for one individual Internet-Draft,
draft-kelley-aipref-training-preference-signal, which uses a reserved robots.txt group (User-Agent: __usage_ai-trainingfollowed byDisallow: /). Anthropic adds that a group naming ClaudeBot takes priority over that signal.
No other company in the table above documents support for these signals as of this writing, so keep the explicit tokens.
A complete robots.txt
This file refuses training at every company listed above while keeping search engines and AI search crawlers allowed. Under RFC 9309, a crawler that finds a group naming it ignores the * group, so the exclusions of the * group are repeated in the named search group.
# robots.txt - https://www.example.com/robots.txt
# Goal: opt out of AI training, stay visible in search and AI answers.
# 1. AI training: refused
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: Bytespider
Disallow: /
# 2. AI search and user-initiated fetches: allowed
# Named groups ignore the * group, so repeat its exclusions here.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-webindexer
User-agent: Amzn-SearchBot
User-agent: MistralAI-Index
User-agent: MistralAI-User
Disallow: /admin/
Allow: /
# 3. Everyone else, including Googlebot, Bingbot and Applebot
User-agent: *
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xml
Adjust it to your goal:
- To stay a source for Gemini answers, remove
Google-Extendedfrom the first group and accept that your content may also be used for Gemini training. - To keep Amazon's products using your pages while refusing training, remove
Amazonbotand addnoarchiveto your pages instead. - For Microsoft, robots.txt is not the lever. If you want to refuse training by Microsoft's models, add the meta tag below, knowing it also keeps the page out of Copilot answers.
<meta name="robots" content="noarchive">
A generic robots meta tag is read by every crawler that supports it, so check what noarchive means for the other engines you care about before applying it site-wide.
The file is explained line by line in block AI training with robots.txt, along with the common RFC 9309 mistakes. To build a variant for your own site, use the robots.txt generator.
Platform settings
If you cannot edit robots.txt directly, your host or CDN may write the rules for you, with its own list of bots.
- Cloudflare classifies AI bots into Search, Agent and Training categories and lets you block each category at the edge, which is enforcement rather than a request. Cloudflare announced the deprecation of the older single Block AI bots setting on September 15, 2026. Details in Cloudflare and AI crawlers.
- Squarespace does not let you edit robots.txt. Its Block known artificial intelligence crawlers checkbox, under Settings > Crawlers, adds a Squarespace-maintained list to robots.txt that mixes training crawlers with agents that are not documented as training crawlers, such as DuckAssistBot, as well as obsolete tokens such as
anthropic-ai. See Squarespace and AI crawlers. - WordPress, Wix, Shopify and Webflow each handle robots.txt differently: see the guides for WordPress, Wix, Shopify and Webflow.
Whatever the platform, open https://your-domain/robots.txt after changing a setting and check that the expected tokens are there and that the file returns HTTP 200. To see which training tokens other sites actually disallow, the weekly AI crawler census tracks the robots.txt files of the top 10,000 sites.
Official sources
- OpenAI, Overview of OpenAI Crawlers: https://developers.openai.com/api/docs/bots
- Google, Google's common crawlers (Google-Extended): https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended
- Google Search Central, AI features and your website: https://developers.google.com/search/docs/appearance/ai-features
- Apple, About Applebot: https://support.apple.com/en-us/119829
- Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler: https://support.claude.com/en/articles/8896518
- IETF, Machine-Readable Training Preference Signal (Internet-Draft): https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/
- Cloudflare, robots.txt setting and Content Signals: https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/
- Meta, Meta Web Crawlers: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- Common Crawl, CCBot: https://commoncrawl.org/ccbot
- Common Crawl, Opt-Out Registry: https://commoncrawl.org/blog/common-crawl-foundation-opt-out-registry
- Amazon, Amazonbot: https://developer.amazon.com/amazonbot
- Mistral AI, Mistral crawlers: https://docs.mistral.ai/robots
- Microsoft Bing, Webmaster Guidelines: https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a
- Microsoft Bing, Robots meta tags and attributes that Bing supports: https://www.bing.com/webmasters/help/robots-meta-tags-and-attributes-that-bing-supports-5198d240
- Perplexity, Perplexity Crawlers: https://docs.perplexity.ai/docs/resources/perplexity-crawlers
- DuckDuckGo, DuckAssistBot: https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot
- Cloudflare, Block AI bots: https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/
- Squarespace, Request that AI models exclude your site: https://support.squarespace.com/hc/en-us/articles/360022347072-Request-that-AI-models-exclude-your-site
- RFC 9309, Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309
Free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
AI crawler blocklists
Ready-to-use files, refreshed hourly, to block AI training (or all AI crawlers): robots.txt, nginx, Apache, Cloudflare and the official IP ranges published by AI companies.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
Related guides
- How to Verify GPTBot, ClaudeBot and PerplexityBot - Official IP Ranges and Reverse DNSConfirm that a GPTBot, ClaudeBot or PerplexityBot request really comes from its operator using official IP ranges or DNS, then block the impostors.
- How to Track AI Crawler Traffic in nginx and Apache Logs - grep, awk and GoAccessCount visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in nginx or Apache logs, per crawler and per page, with a complete script.
- GPTBot vs OAI-SearchBot vs ChatGPT-User - OpenAI's crawlers comparedWhat GPTBot, OAI-SearchBot and ChatGPT-User each do, what blocking them changes for ChatGPT search and training, and how to tell them apart in logs.
- ClaudeBot vs Claude-SearchBot vs Claude-User - Anthropic's crawlers comparedWhat ClaudeBot, Claude-SearchBot and Claude-User do, how each robots.txt rule affects Claude training and search, and why one IP list covers all three bots.