IABIGFR · Français

Platform guides · Updated 2026-11-06 · Markdown version

Block or Allow AI Crawlers on Webflow - robots.txt Rules and the webflow.io Subdomain

Write Webflow robots.txt rules that block GPTBot or ClaudeBot without blocking ChatGPT, Claude and Perplexity, and avoid the trap of the AI bots toggle.

Webflow lets you write your own robots.txt rules in the site's SEO settings, and it also offers toggles that block entire families of crawlers in one click. These toggles are convenient, but the way they group crawlers doesn't match the distinction between training and search: turning off the "AI bots" category also blocks agents that are used to cite you. This guide shows where to write your rules, gives you a ready-to-paste file, and explains how the webflow.io subdomain behaves.

Where the settings are

Everything happens under Site settings > SEO > Indexing. This section includes:

  • the robots.txt rules field;
  • the Staging indexing toggle, which applies to the webflow.io subdomain;
  • the Traffic control toggles, which generate rules for families of crawlers;
  • the Remove sitemap.xml from robots.txt option;
  • the setting for the Content-Signal preferences.

The neighbouring Site settings > SEO > LLMs.txt section is covered further down.

To add your rules:

  1. Open Site settings > SEO > Indexing.
  2. Enter your rules in the robots.txt field.
  3. Click Save, then publish the site.

Changes only take effect after publishing. According to the Help Center, you need a Site plan or a paid Workspace to add a robots.txt file.

A ready-to-paste robots.txt

By default, the robots.txt of a Webflow site on a custom domain often contains only the Sitemap line, which Webflow adds automatically. So you are starting from an almost blank page.

# 1. Model training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Applebot-Extended
User-agent: Google-Extended
Disallow: /

# 2. Search and on-demand retrieval: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# 3. All other crawlers
User-agent: *
Allow: /

Don't add a Sitemap line: Webflow inserts the one for the generated sitemap, unless you turn on Remove sitemap.xml from robots.txt. If you add exclusions to the * group (a thank-you page, a test folder), copy them into group 2: a crawler that finds a group with its own name follows only that group. Remove from group 1 any crawler you are willing to accept, keeping in mind that Google-Extended also covers grounding of answers in the Gemini apps.

The Traffic control toggle trap

The Traffic control toggles automatically generate rules for two families of crawlers, both allowed by default. Webflow's documentation defines them as follows:

  • Search engine crawlers: YandexBot, Googlebot, Bingbot, Baiduspider, OAI-SearchBot, Sogou, DuckDuckBot, MojeekBot, SeznamBot, Qwantify, Yeti, Slurp and 360Spider.
  • AI bots: meta-externalagent, ChatGPT-User, ClaudeBot, PerplexityBot, GPTBot, GoogleOther, Timpibot, SBIntuitionsBot, Bytespider, AI2Bot, Amazon Q, Claude-User, CCBot, Diffbot, YandexGPT, MistralBot, cohere-ai, youBot and NeevaAI.

The "AI bots" category therefore mixes training crawlers (GPTBot, ClaudeBot, CCBot) with search and on-demand retrieval agents (PerplexityBot, ChatGPT-User, Claude-User). Turning this toggle off to block training also removes you from Perplexity and reduces the visits triggered from ChatGPT and Claude. Conversely, Claude-SearchBot, Google-Extended and Applebot-Extended appear in neither list. Webflow itself warns that blocking these families may reduce the site's visibility in both traditional search and AI search.

For the goal of "block training, stay cited", leave both toggles in the allowed position and write your own rules. The documentation doesn't say how the rules generated by these toggles combine with yours: if you use both, read the published file.

The Content-Signal header

Webflow also lets you declare Content-Signal preferences, configured in the same section, to state separately whether your content may be used for training, search and AI-generated answers:

Content-Signal: ai-train=no, search=yes, ai-input=no

Webflow notes that this mechanism is based on a proposed extension to RFC 9309 and is not an accepted standard. Its help center describes it as an HTTP header, but on webflow.com itself (checked in November 2026) the signal appears as a Content-Signal: line inside robots.txt, the form used by Cloudflare's Content Signals Policy, and no such response header is sent. Check your own published robots.txt to see where it lands. No AI company has documented that its crawlers act on it: it complements robots.txt rules without replacing them. Note that ai-input=no expresses a refusal to be used as a source for generated answers, which works against the goal of being visible in AI assistants.

The webflow.io subdomain

Every Webflow site is also published to a staging address on webflow.io, which duplicates your content. To exclude it, switch Staging indexing off under Site settings > SEO > Indexing, then save and publish. Webflow then publishes a separate robots.txt for the subdomain that asks search engines to ignore it; on the subdomains we looked at, it contained User-agent: * followed by Disallow: /, which applies to AI crawlers too. This setting also requires a Site plan or a paid Workspace.

Two practical consequences:

  • your custom rules apply to the custom domain, and the webflow.io subdomain has its own file when Staging indexing is off. The documentation doesn't say which file the subdomain serves when the toggle stays on;
  • as long as the subdomain stays open, crawlers, including those of AI assistants, may find two copies of your pages: turning off its indexing as soon as you go live avoids this duplication.

Webflow-specific limits and pitfalls

  • The file can't be deleted. Once created, the custom robots.txt cannot be removed entirely. To go back to a permissive file, Webflow recommends replacing it with User-agent: * followed by an empty Disallow:, then republishing.
  • Files served by the CDN. Images and documents on a Webflow site are served from Webflow's CDN, not from your domain. Your robots.txt doesn't apply to them, which the documentation confirms.
  • Not for hiding a page. To remove a page from search results, Webflow recommends the Sitemap indexing toggle in the page settings, which adds a noindex tag, rather than a Disallow rule.
  • The Enterprise API. Sites in an Enterprise workspace can also replace or delete the rules through the Webflow Data API; make sure no automation tool overwrites your changes.

llms.txt and bot analytics in Webflow

Two more Webflow features touch AI crawlers, without controlling them:

  • llms.txt upload. Under Site settings > SEO > LLMs.txt, you can upload an llms.txt file (UTF-8, under 100 KB). After publishing, it is served at the root of your custom domain, but not on the webflow.io staging domain. Webflow itself calls the format experimental, with unproven impact. Our llms.txt guide explains what it can and can't do, and the llms.txt generator gives you a starting file.
  • LLM bot insights. On a Team or Enterprise plan with the newest Enterprise Analyze add-on, the Insights > AI discovery tab lists which AI bots requested which pages, based on server logs, and groups them into agent, training, indexing and multi-purpose bots. It is a quick way to confirm that GPTBot stops fetching pages after your rules go live while OAI-SearchBot keeps coming.

Check the result

After publishing, open https://www.example.com/robots.txt, then https://your-site.webflow.io/robots.txt. The first should contain your groups and the Sitemap line, the second a blanket Disallow: / if you turned off Staging indexing. Then test the home page for GPTBot, OAI-SearchBot and PerplexityBot with the IABIG robots.txt tester: the first should be disallowed, the other two allowed. Allow up to 24 hours before crawlers re-read the file (RFC 9309 asks them not to cache it longer than that).

Official sources

Free tools

Related guides