Block or Allow AI Crawlers on Webflow - robots.txt Rules and the webflow.io Subdomain
Write Webflow robots.txt rules that block GPTBot or ClaudeBot without blocking ChatGPT, Claude and Perplexity, and avoid the trap of the AI bots toggle.
Webflow lets you write your own robots.txt rules in the site's SEO settings, and it also offers toggles that block entire families of crawlers in one click. These toggles are convenient, but the way they group crawlers doesn't match the distinction between training and search: turning off the "AI bots" category also blocks agents that are used to cite you. This guide shows where to write your rules, gives you a ready-to-paste file, and explains how the webflow.io subdomain behaves.
Where the settings are
Everything happens under Site settings > SEO > Indexing. This section includes:
- the
robots.txtrules field; - the Staging indexing toggle, which applies to the
webflow.iosubdomain; - the Traffic control toggles, which generate rules for families of crawlers;
- the Remove sitemap.xml from robots.txt option;
- the setting for the
Content-Signalpreferences.
The neighbouring Site settings > SEO > LLMs.txt section is covered further down.
To add your rules:
- Open Site settings > SEO > Indexing.
- Enter your rules in the
robots.txtfield. - Click Save, then publish the site.
Changes only take effect after publishing. According to the Help Center, you need a Site plan or a paid Workspace to add a robots.txt file.
A ready-to-paste robots.txt
By default, the robots.txt of a Webflow site on a custom domain often contains only the Sitemap line, which Webflow adds automatically. So you are starting from an almost blank page.
# 1. Model training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Applebot-Extended
User-agent: Google-Extended
Disallow: /
# 2. Search and on-demand retrieval: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# 3. All other crawlers
User-agent: *
Allow: /
Don't add a Sitemap line: Webflow inserts the one for the generated sitemap, unless you turn on Remove sitemap.xml from robots.txt. If you add exclusions to the * group (a thank-you page, a test folder), copy them into group 2: a crawler that finds a group with its own name follows only that group. Remove from group 1 any crawler you are willing to accept, keeping in mind that Google-Extended also covers grounding of answers in the Gemini apps.
The Traffic control toggle trap
The Traffic control toggles automatically generate rules for two families of crawlers, both allowed by default. Webflow's documentation defines them as follows:
- Search engine crawlers: YandexBot, Googlebot, Bingbot, Baiduspider, OAI-SearchBot, Sogou, DuckDuckBot, MojeekBot, SeznamBot, Qwantify, Yeti, Slurp and 360Spider.
- AI bots: meta-externalagent, ChatGPT-User, ClaudeBot, PerplexityBot, GPTBot, GoogleOther, Timpibot, SBIntuitionsBot, Bytespider, AI2Bot, Amazon Q, Claude-User, CCBot, Diffbot, YandexGPT, MistralBot, cohere-ai, youBot and NeevaAI.
The "AI bots" category therefore mixes training crawlers (GPTBot, ClaudeBot, CCBot) with search and on-demand retrieval agents (PerplexityBot, ChatGPT-User, Claude-User). Turning this toggle off to block training also removes you from Perplexity and reduces the visits triggered from ChatGPT and Claude. Conversely, Claude-SearchBot, Google-Extended and Applebot-Extended appear in neither list. Webflow itself warns that blocking these families may reduce the site's visibility in both traditional search and AI search.
For the goal of "block training, stay cited", leave both toggles in the allowed position and write your own rules. The documentation doesn't say how the rules generated by these toggles combine with yours: if you use both, read the published file.
The Content-Signal header
Webflow also lets you declare Content-Signal preferences, configured in the same section, to state separately whether your content may be used for training, search and AI-generated answers:
Content-Signal: ai-train=no, search=yes, ai-input=no
Webflow notes that this mechanism is based on a proposed extension to RFC 9309 and is not an accepted standard. Its help center describes it as an HTTP header, but on webflow.com itself (checked in November 2026) the signal appears as a Content-Signal: line inside robots.txt, the form used by Cloudflare's Content Signals Policy, and no such response header is sent. Check your own published robots.txt to see where it lands. No AI company has documented that its crawlers act on it: it complements robots.txt rules without replacing them. Note that ai-input=no expresses a refusal to be used as a source for generated answers, which works against the goal of being visible in AI assistants.
The webflow.io subdomain
Every Webflow site is also published to a staging address on webflow.io, which duplicates your content. To exclude it, switch Staging indexing off under Site settings > SEO > Indexing, then save and publish. Webflow then publishes a separate robots.txt for the subdomain that asks search engines to ignore it; on the subdomains we looked at, it contained User-agent: * followed by Disallow: /, which applies to AI crawlers too. This setting also requires a Site plan or a paid Workspace.
Two practical consequences:
- your custom rules apply to the custom domain, and the
webflow.iosubdomain has its own file when Staging indexing is off. The documentation doesn't say which file the subdomain serves when the toggle stays on; - as long as the subdomain stays open, crawlers, including those of AI assistants, may find two copies of your pages: turning off its indexing as soon as you go live avoids this duplication.
Webflow-specific limits and pitfalls
- The file can't be deleted. Once created, the custom
robots.txtcannot be removed entirely. To go back to a permissive file, Webflow recommends replacing it withUser-agent: *followed by an emptyDisallow:, then republishing. - Files served by the CDN. Images and documents on a Webflow site are served from Webflow's CDN, not from your domain. Your
robots.txtdoesn't apply to them, which the documentation confirms. - Not for hiding a page. To remove a page from search results, Webflow recommends the Sitemap indexing toggle in the page settings, which adds a
noindextag, rather than aDisallowrule. - The Enterprise API. Sites in an Enterprise workspace can also replace or delete the rules through the Webflow Data API; make sure no automation tool overwrites your changes.
llms.txt and bot analytics in Webflow
Two more Webflow features touch AI crawlers, without controlling them:
- llms.txt upload. Under Site settings > SEO > LLMs.txt, you can upload an
llms.txtfile (UTF-8, under 100 KB). After publishing, it is served at the root of your custom domain, but not on thewebflow.iostaging domain. Webflow itself calls the format experimental, with unproven impact. Our llms.txt guide explains what it can and can't do, and the llms.txt generator gives you a starting file. - LLM bot insights. On a Team or Enterprise plan with the newest Enterprise Analyze add-on, the Insights > AI discovery tab lists which AI bots requested which pages, based on server logs, and groups them into agent, training, indexing and multi-purpose bots. It is a quick way to confirm that
GPTBotstops fetching pages after your rules go live whileOAI-SearchBotkeeps coming.
Check the result
After publishing, open https://www.example.com/robots.txt, then https://your-site.webflow.io/robots.txt. The first should contain your groups and the Sitemap line, the second a blanket Disallow: / if you turned off Staging indexing. Then test the home page for GPTBot, OAI-SearchBot and PerplexityBot with the IABIG robots.txt tester: the first should be disallowed, the other two allowed. Allow up to 24 hours before crawlers re-read the file (RFC 9309 asks them not to cache it longer than that).
Official sources
- Webflow, Set robots.txt rules: https://help.webflow.com/hc/en-us/articles/41954080897683-Set-robots-txt-rules
- Webflow, Disable search engine indexing: https://help.webflow.com/hc/en-us/articles/33961368603539-Disable-search-engine-indexing
- Webflow, Request to block specific bots from visiting your site: https://help.webflow.com/hc/en-us/articles/41954183592851-Request-to-block-specific-bots-from-visiting-your-site
- Webflow, Upload an llms.txt file to your site: https://help.webflow.com/hc/en-us/articles/43240104183315-Upload-an-llms-txt-file-to-your-site
- Webflow, LLM bot insights: https://help.webflow.com/hc/en-us/articles/51704611767059-LLM-bot-insights
- Webflow, Replace robots.txt (Enterprise API): https://developers.webflow.com/data/reference/enterprise/site-configuration/robots-txt/put
- RFC 9309, Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309
Free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
AI crawler blocklists
Ready-to-use files, refreshed hourly, to block AI training (or all AI crawlers): robots.txt, nginx, Apache, Cloudflare and the official IP ranges published by AI companies.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
Related guides
- How to Block AI Crawlers on IIS - robots.txt, URL Rewrite, Request Filtering and IP RestrictionsOn IIS, publish robots.txt, deny AI training crawlers with a URL Rewrite rule in web.config and learn the limits of request filtering and IP restrictions.
- How to Block AI Crawlers in nginx - User-Agent Rules, Official IP Ranges and Rate LimitsA tested nginx configuration that denies AI training crawlers, lets genuine AI search crawlers through, rate-limits them and writes them to a dedicated log.
- How to Block AI Crawlers in Apache 2.4 - robots.txt, .htaccess, Require not env and IP RangesDeny AI training crawlers in Apache 2.4 with .htaccess, mod_setenvif or mod_rewrite, and only accept AI search crawlers coming from their official IP ranges.
- Block or Allow AI Crawlers on WordPress - Virtual robots.txt, Physical File and PHP FilterWhere to edit robots.txt on WordPress to block GPTBot or ClaudeBot yet stay visible in ChatGPT Search, Claude and Perplexity, by file, PHP filter or plugin.