# Is Cloudflare Blocking AI Crawlers on Your Site - How to Check and Fix It

> Search, Agent and Training policies, Bot Preference Sync, AI Crawl Control, pay per crawl, and how to see what Cloudflare blocks and adjust it to your goals.

Source: https://iabig.com/guides/ai-visibility/cloudflare-ai-crawlers/ — Guides CC BY 4.0, extraits de code MIT

If your site runs through Cloudflare, your `robots.txt` is not the only thing that decides whether ChatGPT, Claude or Perplexity can read your pages. Several Cloudflare settings can block AI crawlers at the network edge, or even rewrite your `robots.txt`. These settings have changed a lot: default blocking of AI crawlers announced in July 2025, Search, Agent and Training policies introduced in July 2026, then further changes on September 15, 2026. This guide describes the current state based on Cloudflare's documentation and blog (checked in early November 2026), and explains how to check and then adjust your configuration. Cloudflare renames these screens often, so treat menu names as a guide and trust what your dashboard shows.

## The mechanisms involved

| Mechanism | Where to find it | Effect |
|---|---|---|
| AI crawler policies (Search, Agent, Training) | **Security** > **Settings**, **Bot traffic** filter, **AI Crawler and Agent access policies** (formerly **Configure AI bot policies**) | Edge blocking by behavior category |
| Bot Preference Sync | Same screen, **Enable Bot Preference Sync** toggle | Adds directives at the top of your `robots.txt` |
| AI Crawl Control | **AI Crawl Control** entry in the zone dashboard | Per-crawler statistics, crawler-by-crawler blocking, charging |
| Bot Fight Mode, Super Bot Fight Mode, WAF rules | **Security** > **Settings**, **Bot traffic** filter; WAF custom rules | General challenges and blocks, not specific to AI |

Cloudflare's September 15, 2026 announcement deprecated the old single **Block AI bots** setting in favor of the three policies, and the old "Managed robots.txt" setting in favor of Bot Preference Sync.

## The Search, Agent and Training policies

Cloudflare classifies crawlers by behavior, and a single crawler can have several:

- **Search**: crawlers that collect or index content to answer later (OAI-SearchBot, Claude-SearchBot, PerplexityBot, but also traditional search engines, because Cloudflare classifies crawlers by what they do rather than by whether they are "AI").
- **Agent**: real-time automated activity on behalf of a person, such as assistants' retrieval crawlers (ChatGPT-User, Claude-User, Perplexity-User) and agents that drive a browser.
- **Training**: crawlers that collect content to train or fine-tune a model (GPTBot, ClaudeBot, and so on), including mixed-purpose crawlers that also do search.

For each category, the options are **Allow**, **Block on pages with ads** (blocking limited to pages where Cloudflare detects advertising) and **Block**. The Training category also offers **Disallow** (called **Disallow AI Training** in Cloudflare's blog).

**The trap introduced on September 15, 2026.** The Block and Block on pages with ads options now also apply to "mixed-use" crawlers that do both search and training. Cloudflare names Googlebot, Bingbot and Applebot. Choosing **Block** for Training can therefore cut your site off from Google Search and Bing. To opt out of training while staying in search engines, choose **Disallow AI Training**. Cloudflare then publishes a no-training preference in `robots.txt` and, with Bot Preference Sync on, blocks training crawlers at the edge, except mixed-use ones. The mixed-use crawlers of operators it labels "Accountable" get through (Cloudflare says Apple, Google and Microsoft meet its criteria), while dedicated training crawlers are blocked; Cloudflare names those of Amazon, Anthropic, Meta and OpenAI. Cloudflare also notes that Bing does not yet read a no-training preference in `robots.txt`: Bingbot honors the `noarchive` meta tag for that today, and Microsoft is targeting robots.txt support for early 2027.

## What may have been turned on without you

For existing domains that had never touched the detailed policies, Cloudflare migrated the old setting as follows:

| Old "Block AI bots" | Search | Training | Agent |
|---|---|---|---|
| Off | Allow | Allow | Allow |
| Block | Allow | Disallow AI Training | Block on pages with ads |
| Block on pages with ads | Allow | Disallow AI Training | Block on pages with ads |

A consequence that is easy to miss: if you had turned on the old blocking, **agents** (ChatGPT-User, Claude-User, Perplexity-User) are now blocked on your pages with ads. When a user asks their assistant to read one of those pages, the request fails.

For a **new domain** added since September 15, 2026, Cloudflare offers two presets at signup, which you can change at any time:

| Setting | Site without ads | Ad-supported site |
|---|---|---|
| Bot Preference Sync | On | On |
| Search | Allow | Allow |
| Training | Allow | Disallow AI Training |
| Agent | Allow | Block on pages with ads |

These presets come from the September 15, 2026 blog post. Cloudflare's documentation page on AI bot policies (last updated November 2, 2026) simply says new domains default to Allow for all three categories, and the July 2026 announcement had described yet another default (Training and Agent blocked on pages with ads). The sources disagree, so check your dashboard rather than assuming.

## The robots.txt managed by Cloudflare

With Bot Preference Sync on and at least one policy set to Block, Block on pages with ads or Disallow, Cloudflare **prepends** directives to your `robots.txt` (or creates one if your origin server has none). Your original content is kept below. If every policy is set to Allow, nothing is added. The added block contains the Content Signals policy, per-crawler `Disallow` lines and start and end markers:

```text
# BEGIN Cloudflare Managed content

User-Agent: *
Content-signal: search=yes, ai-train=no, use=reference
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: GPTBot
Disallow: /

# END Cloudflare Managed Content
```

This excerpt comes from the example in the documentation. Cloudflare keeps the actual list of crawlers up to date, and the marker's wording may vary. `Content-signal` distinguishes three uses: `search` (search index and snippets), `ai-input` (content fed into a model for a real-time answer) and `ai-train` (training). Cloudflare is also testing an optional `use` field (`immediate`, `reference` or `full`, from most to least restrictive) and adds `use=reference` by default when the managed setting is on. Keep in mind that `robots.txt` only expresses a preference. The policies and AI Crawl Control are what actually block.

On the Free plan, a domain with neither its own `robots.txt` nor the managed setting is served only the Content Signals Policy, which defines the categories but expresses no preference.

## AI Crawl Control

AI Crawl Control (formerly **AI Audit**, renamed in August 2025) is available on all plans. Its main tabs:

- **Overview**: a summary of activity per crawler and per operator.
- **Crawlers**: the crawlers seen on the zone, request counts, `robots.txt` violations, and an **Action** column to choose **Allow** or **Block** (and **Charge** for pay per crawl participants).
- **Metrics**: breakdowns by crawler, operator, status code, hostname and path. Outside Enterprise with Bot Management, the analytics window is limited to the last 24 hours.
- **Directives** (renamed from **Robots.txt** in April 2026): the status of your `robots.txt` files and the crawlers requesting disallowed paths.
- **Settings**: the response sent to blocked crawlers (403 or 402 with a plain-text body; customizing it requires a paid plan).

Blocking a crawler in AI Crawl Control creates or edits a WAF custom rule, and these rules run before Cloudflare's other bot products. Outside Enterprise with Bot Management, detection relies on the user agent, so it only catches crawlers that identify themselves. Detection based on Bot Management's detection IDs requires that Enterprise add-on.

## Pay per crawl

Pay per crawl lets you set a per-request price for AI crawlers. A crawler that agrees to pay receives the page (HTTP 200), and the others receive a **402 Payment Required** response with the price. Cloudflare acts as the merchant of record. As of early November 2026, the documentation still describes a closed (private) beta that you join through a sign-up form or, for Enterprise customers, through your account team, and the plan table lists it under Enterprise with Bot Management. Pay per crawl runs after Cloudflare's bot products, so a block by the WAF or by Bot Management takes precedence over charging, and the old Block AI Bots setting had to be turned off for pay per crawl to work.

## A five-minute check

**1. Read the robots.txt served to the public**, not the one on your origin server:

```bash
curl -s https://www.example.com/robots.txt | grep -n -i -E "cloudflare|content-signal|gptbot|claudebot|perplexitybot|oai-searchbot"
```

A `BEGIN Cloudflare` block means Bot Preference Sync (or the old managed robots.txt) is active.

**2. Compare responses by user agent:**

```bash
for ua in "Mozilla/5.0" "OAI-SearchBot/1.4" "GPTBot/1.4" "ChatGPT-User/1.0" "ClaudeBot/1.0" "PerplexityBot/1.0"; do
  printf "%-20s " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "$ua" https://www.example.com/
done
```

A 403 or a challenge page for a given crawler reveals a rule based on the user agent. A 200 does not prove the real crawler gets through. Cloudflare identifies verified bots by more than their name, and a request that imitates a user agent is not treated as the genuine crawler.

**3. In the dashboard**, open **Security** > **Settings**, filter on **Bot traffic**, and note the value of the three policies and the state of **Enable Bot Preference Sync** and **Bot fight mode**.

**4. In AI Crawl Control** > **Crawlers**, check the Action column crawler by crawler, then look at the breakdown of status codes in **Metrics**. Many 403s for OAI-SearchBot or Claude-SearchBot point to a block.

**5. In Security > Analytics**, on the **Events** tab, look for blocked or challenged requests and the service responsible (WAF rule, Bot Fight Mode, and so on). Also remember manually created WAF rules and IP access rules, which block independently of the AI settings.

## Adjusting to your goal

| Goal | Search | Training | Agent |
|---|---|---|---|
| Be found and cited by assistants, no restrictions | Allow | Allow | Allow |
| Be found and cited, but opt out of training | Allow | Disallow AI Training | Allow |
| Ad-supported site, protect pages with ads | Allow | Disallow AI Training | Block on pages with ads |
| Stay out of AI services entirely | Block | Block | Block |

The last row also blocks Googlebot, Bingbot and Applebot, so your site disappears from traditional search engines too. If visibility is your goal, do not block **Search** and **Agent**. They are what let you appear in answers from ChatGPT, Claude or Perplexity and get read when a user cites your page. For a case-by-case policy (one crawler allowed, another not), turn off Bot Preference Sync, set crawlers individually in AI Crawl Control and maintain your `robots.txt` by hand.

After each change, repeat checks 1 and 2. The documentation says the managed content appears in `robots.txt` within a few seconds, with no cache purge needed. To check your site from the outside, use the [IABIG checker](/). For the crawlers behind each category, see the [AI crawler user agents list](/guides/ai-crawlers/ai-crawler-user-agents-list/).

## Sources

- [Cloudflare blog, Have it both ways (September 15, 2026)](https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/)
- [Cloudflare blog, Say it once, introducing Bot Preference Sync (August 2026)](https://blog.cloudflare.com/bot-preference-sync/)
- [Cloudflare blog, Content Independence Day (July 1, 2025)](https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/)
- [Cloudflare docs, Block AI Bots](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/)
- [Cloudflare docs, robots.txt setting](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/)
- [Cloudflare docs, AI bots (classification)](https://developers.cloudflare.com/bots/concepts/bot/)
- [Cloudflare docs, Verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/)
- [Cloudflare docs, AI Crawl Control](https://developers.cloudflare.com/ai-crawl-control/)
- [Cloudflare docs, AI Crawl Control, Get started](https://developers.cloudflare.com/ai-crawl-control/get-started/)
- [Cloudflare docs, Manage AI crawlers](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/)
- [Cloudflare docs, AI Crawl Control changelog](https://developers.cloudflare.com/ai-crawl-control/changelog/)
- [Cloudflare docs, AI Crawl Control with Cloudflare Bots](https://developers.cloudflare.com/ai-crawl-control/configuration/ai-crawl-control-with-bots/)
- [Cloudflare docs, What is Pay Per Crawl](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/)
- [Cloudflare docs, Bot reference](https://developers.cloudflare.com/ai-crawl-control/reference/bots/)
- [Cloudflare docs, Bot Fight Mode](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/)
