# Block or Allow AI Crawlers on Webflow - robots.txt Rules and the webflow.io Subdomain

> Write Webflow robots.txt rules that block GPTBot or ClaudeBot without blocking ChatGPT, Claude and Perplexity, and avoid the trap of the AI bots toggle.

Source: https://iabig.com/guides/platforms/webflow-ai-crawlers/ — Guides CC BY 4.0, extraits de code MIT

Webflow lets you write your own `robots.txt` rules in the site's SEO settings, and it also offers toggles that block entire families of crawlers in one click. These toggles are convenient, but the way they group crawlers doesn't match the distinction between training and search: turning off the "AI bots" category also blocks agents that are used to cite you. This guide shows where to write your rules, gives you a ready-to-paste file, and explains how the `webflow.io` subdomain behaves.

## Where the settings are

Everything happens under **Site settings > SEO > Indexing**. This section includes:

- the `robots.txt` rules field;
- the **Staging indexing** toggle, which applies to the `webflow.io` subdomain;
- the **Traffic control** toggles, which generate rules for families of crawlers;
- the **Remove sitemap.xml from robots.txt** option;
- the setting for the `Content-Signal` preferences.

The neighbouring **Site settings > SEO > LLMs.txt** section is covered further down.

To add your rules:

1. Open **Site settings > SEO > Indexing**.
2. Enter your rules in the `robots.txt` field.
3. Click **Save**, then publish the site.

Changes only take effect after publishing. According to the Help Center, you need a Site plan or a paid Workspace to add a `robots.txt` file.

## A ready-to-paste robots.txt

By default, the `robots.txt` of a Webflow site on a custom domain often contains only the `Sitemap` line, which Webflow adds automatically. So you are starting from an almost blank page.

```text
# 1. Model training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Applebot-Extended
User-agent: Google-Extended
Disallow: /

# 2. Search and on-demand retrieval: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# 3. All other crawlers
User-agent: *
Allow: /
```

Don't add a `Sitemap` line: Webflow inserts the one for the generated sitemap, unless you turn on **Remove sitemap.xml from robots.txt**. If you add exclusions to the `*` group (a thank-you page, a test folder), copy them into group 2: a crawler that finds a group with its own name follows only that group. Remove from group 1 any crawler you are willing to accept, keeping in mind that `Google-Extended` also covers grounding of answers in the Gemini apps.

## The Traffic control toggle trap

The **Traffic control** toggles automatically generate rules for two families of crawlers, both allowed by default. Webflow's documentation defines them as follows:

- **Search engine crawlers**: YandexBot, Googlebot, Bingbot, Baiduspider, OAI-SearchBot, Sogou, DuckDuckBot, MojeekBot, SeznamBot, Qwantify, Yeti, Slurp and 360Spider.
- **AI bots**: meta-externalagent, ChatGPT-User, ClaudeBot, PerplexityBot, GPTBot, GoogleOther, Timpibot, SBIntuitionsBot, Bytespider, AI2Bot, Amazon Q, Claude-User, CCBot, Diffbot, YandexGPT, MistralBot, cohere-ai, youBot and NeevaAI.

The "AI bots" category therefore mixes training crawlers (`GPTBot`, `ClaudeBot`, `CCBot`) with search and on-demand retrieval agents (`PerplexityBot`, `ChatGPT-User`, `Claude-User`). Turning this toggle off to block training also removes you from Perplexity and reduces the visits triggered from ChatGPT and Claude. Conversely, `Claude-SearchBot`, `Google-Extended` and `Applebot-Extended` appear in neither list. Webflow itself warns that blocking these families may reduce the site's visibility in both traditional search and AI search.

For the goal of "block training, stay cited", leave both toggles in the allowed position and write your own rules. The documentation doesn't say how the rules generated by these toggles combine with yours: if you use both, read the published file.

## The Content-Signal header

Webflow also lets you declare `Content-Signal` preferences, configured in the same section, to state separately whether your content may be used for training, search and AI-generated answers:

```text
Content-Signal: ai-train=no, search=yes, ai-input=no
```

Webflow notes that this mechanism is based on a proposed extension to RFC 9309 and is not an accepted standard. Its help center describes it as an HTTP header, but on webflow.com itself (checked in November 2026) the signal appears as a `Content-Signal:` line inside `robots.txt`, the form used by Cloudflare's Content Signals Policy, and no such response header is sent. Check your own published `robots.txt` to see where it lands. No AI company has documented that its crawlers act on it: it complements `robots.txt` rules without replacing them. Note that `ai-input=no` expresses a refusal to be used as a source for generated answers, which works against the goal of being visible in AI assistants.

## The webflow.io subdomain

Every Webflow site is also published to a staging address on `webflow.io`, which duplicates your content. To exclude it, switch **Staging indexing** off under **Site settings > SEO > Indexing**, then save and publish. Webflow then publishes a separate `robots.txt` for the subdomain that asks search engines to ignore it; on the subdomains we looked at, it contained `User-agent: *` followed by `Disallow: /`, which applies to AI crawlers too. This setting also requires a Site plan or a paid Workspace.

Two practical consequences:

- your custom rules apply to the custom domain, and the `webflow.io` subdomain has its own file when **Staging indexing** is off. The documentation doesn't say which file the subdomain serves when the toggle stays on;
- as long as the subdomain stays open, crawlers, including those of AI assistants, may find two copies of your pages: turning off its indexing as soon as you go live avoids this duplication.

## Webflow-specific limits and pitfalls

- **The file can't be deleted.** Once created, the custom `robots.txt` cannot be removed entirely. To go back to a permissive file, Webflow recommends replacing it with `User-agent: *` followed by an empty `Disallow:`, then republishing.
- **Files served by the CDN.** Images and documents on a Webflow site are served from Webflow's CDN, not from your domain. Your `robots.txt` doesn't apply to them, which the documentation confirms.
- **Not for hiding a page.** To remove a page from search results, Webflow recommends the **Sitemap indexing** toggle in the page settings, which adds a `noindex` tag, rather than a `Disallow` rule.
- **The Enterprise API.** Sites in an Enterprise workspace can also replace or delete the rules through the Webflow Data API; make sure no automation tool overwrites your changes.

## llms.txt and bot analytics in Webflow

Two more Webflow features touch AI crawlers, without controlling them:

- **llms.txt upload.** Under **Site settings > SEO > LLMs.txt**, you can upload an `llms.txt` file (UTF-8, under 100 KB). After publishing, it is served at the root of your custom domain, but not on the `webflow.io` staging domain. Webflow itself calls the format experimental, with unproven impact. Our [llms.txt guide](/guides/ai-visibility/llms-txt-guide/) explains what it can and can't do, and the [llms.txt generator](/tools/llms-txt-generator/) gives you a starting file.
- **LLM bot insights.** On a Team or Enterprise plan with the newest Enterprise Analyze add-on, the **Insights > AI discovery** tab lists which AI bots requested which pages, based on server logs, and groups them into agent, training, indexing and multi-purpose bots. It is a quick way to confirm that `GPTBot` stops fetching pages after your rules go live while `OAI-SearchBot` keeps coming.

## Check the result

After publishing, open `https://www.example.com/robots.txt`, then `https://your-site.webflow.io/robots.txt`. The first should contain your groups and the `Sitemap` line, the second a blanket `Disallow: /` if you turned off **Staging indexing**. Then test the home page for `GPTBot`, `OAI-SearchBot` and `PerplexityBot` with the [IABIG robots.txt tester](/tools/robots-txt-tester/): the first should be disallowed, the other two allowed. Allow up to 24 hours before crawlers re-read the file (RFC 9309 asks them not to cache it longer than that).

## Official sources

- Webflow, Set robots.txt rules: <https://help.webflow.com/hc/en-us/articles/41954080897683-Set-robots-txt-rules>
- Webflow, Disable search engine indexing: <https://help.webflow.com/hc/en-us/articles/33961368603539-Disable-search-engine-indexing>
- Webflow, Request to block specific bots from visiting your site: <https://help.webflow.com/hc/en-us/articles/41954183592851-Request-to-block-specific-bots-from-visiting-your-site>
- Webflow, Upload an llms.txt file to your site: <https://help.webflow.com/hc/en-us/articles/43240104183315-Upload-an-llms-txt-file-to-your-site>
- Webflow, LLM bot insights: <https://help.webflow.com/hc/en-us/articles/51704611767059-LLM-bot-insights>
- Webflow, Replace robots.txt (Enterprise API): <https://developers.webflow.com/data/reference/enterprise/site-configuration/robots-txt/put>
- RFC 9309, Robots Exclusion Protocol: <https://www.rfc-editor.org/rfc/rfc9309>
