IABIGFR · Français

AI visibility · Updated 2026-11-06 · Markdown version

Is Cloudflare Blocking AI Crawlers on Your Site - How to Check and Fix It

Search, Agent and Training policies, Bot Preference Sync, AI Crawl Control, pay per crawl, and how to see what Cloudflare blocks and adjust it to your goals.

If your site runs through Cloudflare, your robots.txt is not the only thing that decides whether ChatGPT, Claude or Perplexity can read your pages. Several Cloudflare settings can block AI crawlers at the network edge, or even rewrite your robots.txt. These settings have changed a lot: default blocking of AI crawlers announced in July 2025, Search, Agent and Training policies introduced in July 2026, then further changes on September 15, 2026. This guide describes the current state based on Cloudflare's documentation and blog (checked in early November 2026), and explains how to check and then adjust your configuration. Cloudflare renames these screens often, so treat menu names as a guide and trust what your dashboard shows.

The mechanisms involved

Mechanism Where to find it Effect
AI crawler policies (Search, Agent, Training) Security > Settings, Bot traffic filter, AI Crawler and Agent access policies (formerly Configure AI bot policies) Edge blocking by behavior category
Bot Preference Sync Same screen, Enable Bot Preference Sync toggle Adds directives at the top of your robots.txt
AI Crawl Control AI Crawl Control entry in the zone dashboard Per-crawler statistics, crawler-by-crawler blocking, charging
Bot Fight Mode, Super Bot Fight Mode, WAF rules Security > Settings, Bot traffic filter; WAF custom rules General challenges and blocks, not specific to AI

Cloudflare's September 15, 2026 announcement deprecated the old single Block AI bots setting in favor of the three policies, and the old "Managed robots.txt" setting in favor of Bot Preference Sync.

The Search, Agent and Training policies

Cloudflare classifies crawlers by behavior, and a single crawler can have several:

  • Search: crawlers that collect or index content to answer later (OAI-SearchBot, Claude-SearchBot, PerplexityBot, but also traditional search engines, because Cloudflare classifies crawlers by what they do rather than by whether they are "AI").
  • Agent: real-time automated activity on behalf of a person, such as assistants' retrieval crawlers (ChatGPT-User, Claude-User, Perplexity-User) and agents that drive a browser.
  • Training: crawlers that collect content to train or fine-tune a model (GPTBot, ClaudeBot, and so on), including mixed-purpose crawlers that also do search.

For each category, the options are Allow, Block on pages with ads (blocking limited to pages where Cloudflare detects advertising) and Block. The Training category also offers Disallow (called Disallow AI Training in Cloudflare's blog).

The trap introduced on September 15, 2026. The Block and Block on pages with ads options now also apply to "mixed-use" crawlers that do both search and training. Cloudflare names Googlebot, Bingbot and Applebot. Choosing Block for Training can therefore cut your site off from Google Search and Bing. To opt out of training while staying in search engines, choose Disallow AI Training. Cloudflare then publishes a no-training preference in robots.txt and, with Bot Preference Sync on, blocks training crawlers at the edge, except mixed-use ones. The mixed-use crawlers of operators it labels "Accountable" get through (Cloudflare says Apple, Google and Microsoft meet its criteria), while dedicated training crawlers are blocked; Cloudflare names those of Amazon, Anthropic, Meta and OpenAI. Cloudflare also notes that Bing does not yet read a no-training preference in robots.txt: Bingbot honors the noarchive meta tag for that today, and Microsoft is targeting robots.txt support for early 2027.

What may have been turned on without you

For existing domains that had never touched the detailed policies, Cloudflare migrated the old setting as follows:

Old "Block AI bots" Search Training Agent
Off Allow Allow Allow
Block Allow Disallow AI Training Block on pages with ads
Block on pages with ads Allow Disallow AI Training Block on pages with ads

A consequence that is easy to miss: if you had turned on the old blocking, agents (ChatGPT-User, Claude-User, Perplexity-User) are now blocked on your pages with ads. When a user asks their assistant to read one of those pages, the request fails.

For a new domain added since September 15, 2026, Cloudflare offers two presets at signup, which you can change at any time:

Setting Site without ads Ad-supported site
Bot Preference Sync On On
Search Allow Allow
Training Allow Disallow AI Training
Agent Allow Block on pages with ads

These presets come from the September 15, 2026 blog post. Cloudflare's documentation page on AI bot policies (last updated November 2, 2026) simply says new domains default to Allow for all three categories, and the July 2026 announcement had described yet another default (Training and Agent blocked on pages with ads). The sources disagree, so check your dashboard rather than assuming.

The robots.txt managed by Cloudflare

With Bot Preference Sync on and at least one policy set to Block, Block on pages with ads or Disallow, Cloudflare prepends directives to your robots.txt (or creates one if your origin server has none). Your original content is kept below. If every policy is set to Allow, nothing is added. The added block contains the Content Signals policy, per-crawler Disallow lines and start and end markers:

# BEGIN Cloudflare Managed content

User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: GPTBot
Disallow: /

# END Cloudflare Managed Content

This excerpt comes from the example in the documentation. Cloudflare keeps the actual list of crawlers up to date, and the marker's wording may vary. Content-signal distinguishes three uses: search (search index and snippets), ai-input (content fed into a model for a real-time answer) and ai-train (training). Cloudflare is also testing an optional use field (immediate, reference or full, from most to least restrictive) and adds use=reference by default when the managed setting is on. Keep in mind that robots.txt only expresses a preference. The policies and AI Crawl Control are what actually block.

On the Free plan, a domain with neither its own robots.txt nor the managed setting is served only the Content Signals Policy, which defines the categories but expresses no preference.

AI Crawl Control

AI Crawl Control (formerly AI Audit, renamed in August 2025) is available on all plans. Its main tabs:

  • Overview: a summary of activity per crawler and per operator.
  • Crawlers: the crawlers seen on the zone, request counts, robots.txt violations, and an Action column to choose Allow or Block (and Charge for pay per crawl participants).
  • Metrics: breakdowns by crawler, operator, status code, hostname and path. Outside Enterprise with Bot Management, the analytics window is limited to the last 24 hours.
  • Directives (renamed from Robots.txt in April 2026): the status of your robots.txt files and the crawlers requesting disallowed paths.
  • Settings: the response sent to blocked crawlers (403 or 402 with a plain-text body; customizing it requires a paid plan).

Blocking a crawler in AI Crawl Control creates or edits a WAF custom rule, and these rules run before Cloudflare's other bot products. Outside Enterprise with Bot Management, detection relies on the user agent, so it only catches crawlers that identify themselves. Detection based on Bot Management's detection IDs requires that Enterprise add-on.

Pay per crawl

Pay per crawl lets you set a per-request price for AI crawlers. A crawler that agrees to pay receives the page (HTTP 200), and the others receive a 402 Payment Required response with the price. Cloudflare acts as the merchant of record. As of early November 2026, the documentation still describes a closed (private) beta that you join through a sign-up form or, for Enterprise customers, through your account team, and the plan table lists it under Enterprise with Bot Management. Pay per crawl runs after Cloudflare's bot products, so a block by the WAF or by Bot Management takes precedence over charging, and the old Block AI Bots setting had to be turned off for pay per crawl to work.

A five-minute check

1. Read the robots.txt served to the public, not the one on your origin server:

curl -s https://www.example.com/robots.txt | grep -n -i -E "cloudflare|content-signal|gptbot|claudebot|perplexitybot|oai-searchbot"

A BEGIN Cloudflare block means Bot Preference Sync (or the old managed robots.txt) is active.

2. Compare responses by user agent:

for ua in "Mozilla/5.0" "OAI-SearchBot/1.4" "GPTBot/1.4" "ChatGPT-User/1.0" "ClaudeBot/1.0" "PerplexityBot/1.0"; do
  printf "%-20s " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "$ua" https://www.example.com/
done

A 403 or a challenge page for a given crawler reveals a rule based on the user agent. A 200 does not prove the real crawler gets through. Cloudflare identifies verified bots by more than their name, and a request that imitates a user agent is not treated as the genuine crawler.

3. In the dashboard, open Security > Settings, filter on Bot traffic, and note the value of the three policies and the state of Enable Bot Preference Sync and Bot fight mode.

4. In AI Crawl Control > Crawlers, check the Action column crawler by crawler, then look at the breakdown of status codes in Metrics. Many 403s for OAI-SearchBot or Claude-SearchBot point to a block.

5. In Security > Analytics, on the Events tab, look for blocked or challenged requests and the service responsible (WAF rule, Bot Fight Mode, and so on). Also remember manually created WAF rules and IP access rules, which block independently of the AI settings.

Adjusting to your goal

Goal Search Training Agent
Be found and cited by assistants, no restrictions Allow Allow Allow
Be found and cited, but opt out of training Allow Disallow AI Training Allow
Ad-supported site, protect pages with ads Allow Disallow AI Training Block on pages with ads
Stay out of AI services entirely Block Block Block

The last row also blocks Googlebot, Bingbot and Applebot, so your site disappears from traditional search engines too. If visibility is your goal, do not block Search and Agent. They are what let you appear in answers from ChatGPT, Claude or Perplexity and get read when a user cites your page. For a case-by-case policy (one crawler allowed, another not), turn off Bot Preference Sync, set crawlers individually in AI Crawl Control and maintain your robots.txt by hand.

After each change, repeat checks 1 and 2. The documentation says the managed content appears in robots.txt within a few seconds, with no cache purge needed. To check your site from the outside, use the IABIG checker. For the crawlers behind each category, see the AI crawler user agents list.

Sources

Free tools

Related guides