Is Cloudflare Blocking AI Crawlers on Your Site - How to Check and Fix It
Search, Agent and Training policies, Bot Preference Sync, AI Crawl Control, pay per crawl, and how to see what Cloudflare blocks and adjust it to your goals.
If your site runs through Cloudflare, your robots.txt is not the only thing that decides whether ChatGPT, Claude or Perplexity can read your pages. Several Cloudflare settings can block AI crawlers at the network edge, or even rewrite your robots.txt. These settings have changed a lot: default blocking of AI crawlers announced in July 2025, Search, Agent and Training policies introduced in July 2026, then further changes on September 15, 2026. This guide describes the current state based on Cloudflare's documentation and blog (checked in early November 2026), and explains how to check and then adjust your configuration. Cloudflare renames these screens often, so treat menu names as a guide and trust what your dashboard shows.
The mechanisms involved
| Mechanism | Where to find it | Effect |
|---|---|---|
| AI crawler policies (Search, Agent, Training) | Security > Settings, Bot traffic filter, AI Crawler and Agent access policies (formerly Configure AI bot policies) | Edge blocking by behavior category |
| Bot Preference Sync | Same screen, Enable Bot Preference Sync toggle | Adds directives at the top of your robots.txt |
| AI Crawl Control | AI Crawl Control entry in the zone dashboard | Per-crawler statistics, crawler-by-crawler blocking, charging |
| Bot Fight Mode, Super Bot Fight Mode, WAF rules | Security > Settings, Bot traffic filter; WAF custom rules | General challenges and blocks, not specific to AI |
Cloudflare's September 15, 2026 announcement deprecated the old single Block AI bots setting in favor of the three policies, and the old "Managed robots.txt" setting in favor of Bot Preference Sync.
The Search, Agent and Training policies
Cloudflare classifies crawlers by behavior, and a single crawler can have several:
- Search: crawlers that collect or index content to answer later (OAI-SearchBot, Claude-SearchBot, PerplexityBot, but also traditional search engines, because Cloudflare classifies crawlers by what they do rather than by whether they are "AI").
- Agent: real-time automated activity on behalf of a person, such as assistants' retrieval crawlers (ChatGPT-User, Claude-User, Perplexity-User) and agents that drive a browser.
- Training: crawlers that collect content to train or fine-tune a model (GPTBot, ClaudeBot, and so on), including mixed-purpose crawlers that also do search.
For each category, the options are Allow, Block on pages with ads (blocking limited to pages where Cloudflare detects advertising) and Block. The Training category also offers Disallow (called Disallow AI Training in Cloudflare's blog).
The trap introduced on September 15, 2026. The Block and Block on pages with ads options now also apply to "mixed-use" crawlers that do both search and training. Cloudflare names Googlebot, Bingbot and Applebot. Choosing Block for Training can therefore cut your site off from Google Search and Bing. To opt out of training while staying in search engines, choose Disallow AI Training. Cloudflare then publishes a no-training preference in robots.txt and, with Bot Preference Sync on, blocks training crawlers at the edge, except mixed-use ones. The mixed-use crawlers of operators it labels "Accountable" get through (Cloudflare says Apple, Google and Microsoft meet its criteria), while dedicated training crawlers are blocked; Cloudflare names those of Amazon, Anthropic, Meta and OpenAI. Cloudflare also notes that Bing does not yet read a no-training preference in robots.txt: Bingbot honors the noarchive meta tag for that today, and Microsoft is targeting robots.txt support for early 2027.
What may have been turned on without you
For existing domains that had never touched the detailed policies, Cloudflare migrated the old setting as follows:
| Old "Block AI bots" | Search | Training | Agent |
|---|---|---|---|
| Off | Allow | Allow | Allow |
| Block | Allow | Disallow AI Training | Block on pages with ads |
| Block on pages with ads | Allow | Disallow AI Training | Block on pages with ads |
A consequence that is easy to miss: if you had turned on the old blocking, agents (ChatGPT-User, Claude-User, Perplexity-User) are now blocked on your pages with ads. When a user asks their assistant to read one of those pages, the request fails.
For a new domain added since September 15, 2026, Cloudflare offers two presets at signup, which you can change at any time:
| Setting | Site without ads | Ad-supported site |
|---|---|---|
| Bot Preference Sync | On | On |
| Search | Allow | Allow |
| Training | Allow | Disallow AI Training |
| Agent | Allow | Block on pages with ads |
These presets come from the September 15, 2026 blog post. Cloudflare's documentation page on AI bot policies (last updated November 2, 2026) simply says new domains default to Allow for all three categories, and the July 2026 announcement had described yet another default (Training and Agent blocked on pages with ads). The sources disagree, so check your dashboard rather than assuming.
The robots.txt managed by Cloudflare
With Bot Preference Sync on and at least one policy set to Block, Block on pages with ads or Disallow, Cloudflare prepends directives to your robots.txt (or creates one if your origin server has none). Your original content is kept below. If every policy is set to Allow, nothing is added. The added block contains the Content Signals policy, per-crawler Disallow lines and start and end markers:
# BEGIN Cloudflare Managed content
User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: GPTBot
Disallow: /
# END Cloudflare Managed Content
This excerpt comes from the example in the documentation. Cloudflare keeps the actual list of crawlers up to date, and the marker's wording may vary. Content-signal distinguishes three uses: search (search index and snippets), ai-input (content fed into a model for a real-time answer) and ai-train (training). Cloudflare is also testing an optional use field (immediate, reference or full, from most to least restrictive) and adds use=reference by default when the managed setting is on. Keep in mind that robots.txt only expresses a preference. The policies and AI Crawl Control are what actually block.
On the Free plan, a domain with neither its own robots.txt nor the managed setting is served only the Content Signals Policy, which defines the categories but expresses no preference.
AI Crawl Control
AI Crawl Control (formerly AI Audit, renamed in August 2025) is available on all plans. Its main tabs:
- Overview: a summary of activity per crawler and per operator.
- Crawlers: the crawlers seen on the zone, request counts,
robots.txtviolations, and an Action column to choose Allow or Block (and Charge for pay per crawl participants). - Metrics: breakdowns by crawler, operator, status code, hostname and path. Outside Enterprise with Bot Management, the analytics window is limited to the last 24 hours.
- Directives (renamed from Robots.txt in April 2026): the status of your
robots.txtfiles and the crawlers requesting disallowed paths. - Settings: the response sent to blocked crawlers (403 or 402 with a plain-text body; customizing it requires a paid plan).
Blocking a crawler in AI Crawl Control creates or edits a WAF custom rule, and these rules run before Cloudflare's other bot products. Outside Enterprise with Bot Management, detection relies on the user agent, so it only catches crawlers that identify themselves. Detection based on Bot Management's detection IDs requires that Enterprise add-on.
Pay per crawl
Pay per crawl lets you set a per-request price for AI crawlers. A crawler that agrees to pay receives the page (HTTP 200), and the others receive a 402 Payment Required response with the price. Cloudflare acts as the merchant of record. As of early November 2026, the documentation still describes a closed (private) beta that you join through a sign-up form or, for Enterprise customers, through your account team, and the plan table lists it under Enterprise with Bot Management. Pay per crawl runs after Cloudflare's bot products, so a block by the WAF or by Bot Management takes precedence over charging, and the old Block AI Bots setting had to be turned off for pay per crawl to work.
A five-minute check
1. Read the robots.txt served to the public, not the one on your origin server:
curl -s https://www.example.com/robots.txt | grep -n -i -E "cloudflare|content-signal|gptbot|claudebot|perplexitybot|oai-searchbot"
A BEGIN Cloudflare block means Bot Preference Sync (or the old managed robots.txt) is active.
2. Compare responses by user agent:
for ua in "Mozilla/5.0" "OAI-SearchBot/1.4" "GPTBot/1.4" "ChatGPT-User/1.0" "ClaudeBot/1.0" "PerplexityBot/1.0"; do
printf "%-20s " "$ua"
curl -s -o /dev/null -w "%{http_code}\n" -A "$ua" https://www.example.com/
done
A 403 or a challenge page for a given crawler reveals a rule based on the user agent. A 200 does not prove the real crawler gets through. Cloudflare identifies verified bots by more than their name, and a request that imitates a user agent is not treated as the genuine crawler.
3. In the dashboard, open Security > Settings, filter on Bot traffic, and note the value of the three policies and the state of Enable Bot Preference Sync and Bot fight mode.
4. In AI Crawl Control > Crawlers, check the Action column crawler by crawler, then look at the breakdown of status codes in Metrics. Many 403s for OAI-SearchBot or Claude-SearchBot point to a block.
5. In Security > Analytics, on the Events tab, look for blocked or challenged requests and the service responsible (WAF rule, Bot Fight Mode, and so on). Also remember manually created WAF rules and IP access rules, which block independently of the AI settings.
Adjusting to your goal
| Goal | Search | Training | Agent |
|---|---|---|---|
| Be found and cited by assistants, no restrictions | Allow | Allow | Allow |
| Be found and cited, but opt out of training | Allow | Disallow AI Training | Allow |
| Ad-supported site, protect pages with ads | Allow | Disallow AI Training | Block on pages with ads |
| Stay out of AI services entirely | Block | Block | Block |
The last row also blocks Googlebot, Bingbot and Applebot, so your site disappears from traditional search engines too. If visibility is your goal, do not block Search and Agent. They are what let you appear in answers from ChatGPT, Claude or Perplexity and get read when a user cites your page. For a case-by-case policy (one crawler allowed, another not), turn off Bot Preference Sync, set crawlers individually in AI Crawl Control and maintain your robots.txt by hand.
After each change, repeat checks 1 and 2. The documentation says the managed content appears in robots.txt within a few seconds, with no cache purge needed. To check your site from the outside, use the IABIG checker. For the crawlers behind each category, see the AI crawler user agents list.
Sources
- Cloudflare blog, Have it both ways (September 15, 2026)
- Cloudflare blog, Say it once, introducing Bot Preference Sync (August 2026)
- Cloudflare blog, Content Independence Day (July 1, 2025)
- Cloudflare docs, Block AI Bots
- Cloudflare docs, robots.txt setting
- Cloudflare docs, AI bots (classification)
- Cloudflare docs, Verified bots
- Cloudflare docs, AI Crawl Control
- Cloudflare docs, AI Crawl Control, Get started
- Cloudflare docs, Manage AI crawlers
- Cloudflare docs, AI Crawl Control changelog
- Cloudflare docs, AI Crawl Control with Cloudflare Bots
- Cloudflare docs, What is Pay Per Crawl
- Cloudflare docs, Bot reference
- Cloudflare docs, Bot Fight Mode
Free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
AI crawler blocklists
Ready-to-use files, refreshed hourly, to block AI training (or all AI crawlers): robots.txt, nginx, Apache, Cloudflare and the official IP ranges published by AI companies.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
Related guides
- llms.txt - Exact Format, Examples and Real Adoption by AI CompaniesThe llms.txt file as specified on llmstxt.org, with examples for a business site and a documentation site, and what AI companies actually confirm about it.
- Is Your React, Vue or Angular Site Invisible to AI - Why and How to Fix ItWhy ChatGPT, Claude or Perplexity may not see a JavaScript-rendered site, how to test yours with curl, and which SSR or static generation fixes to apply.