Block or Allow AI Crawlers on Squarespace - What the Crawlers Setting Really Does
Squarespace doesn't let you edit robots.txt. What its AI crawler checkbox blocks, what it lets through, and how to check the file that is actually served.
Squarespace is the most locked-down of the major platforms on this point: the robots.txt file is shared by all sites and cannot be edited. The only lever on offer is a checkbox that adds a list of AI crawlers, chosen by Squarespace, to the file. This guide details exactly what that list covers and what it doesn't, why it partly fits the goal of "block training, stay visible in search", and how to check the file once the setting is applied.
No robots.txt editing
Squarespace's documentation is explicit: every Squarespace site uses the same robots.txt file, and users can neither access nor modify it. Squarespace justifies this as following SEO best practices. Among other things, the file excludes /config/ (the admin login page), /api/ (which Squarespace says blocks its analytics tracking cookie), /search, and URLs that duplicate content, such as lists filtered by author, tag or month. Squarespace points to squarespace.com/robots.txt as the reference copy of the full file.
Two settings affect this file, both in the same panel:
- Block Search engine crawlers, which asks search engines not to index the site by adding a
Disallow: /rule. It removes the site from search engines: don't check it if you want visibility, including in AI assistants. - Block known artificial intelligence crawlers, which adds an exclusion request for a list of AI crawlers.
There is no finer-grained setting. Squarespace states that you cannot ask AI crawlers to crawl only certain pages.
Where to find the setting
The path Squarespace gives is:
- Open the Settings panel.
- Click Crawlers.
- Check Block known artificial intelligence crawlers.
The box is unchecked by default. Squarespace explains this choice by its concern about reducing sites' traffic by excluding them from the answers and cited sources of AI assistants.
What the checkbox blocks exactly
According to Squarespace's help article (updated January 8, 2026), checking the box asks the following crawlers not to crawl the site:
AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended,
Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler,
DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image,
GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent,
MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot
Squarespace says this list may grow, and that you can suggest a missing crawler through a feature request.
Measured against the goal of this site, the list reads as follows:
- The main training crawlers are on it:
GPTBot,ClaudeBot,CCBot,Meta-ExternalAgent(case doesn't matter in the crawler name),Google-ExtendedandApplebot-Extended.MistralAI-Trainingis not. - The AI assistants' search crawlers are not on it:
OAI-SearchBot,ChatGPT-User,Claude-SearchBot,Claude-User,PerplexityBotandPerplexity-Userremain subject to the general*group, so they are allowed on public pages. - Some blocked crawlers are also used to cite you. DuckDuckGo presents
DuckAssistBotas the crawler behind its AI-assisted answers and states that its data is not used to train models.Google-Extendedcovers grounding of answers in the Gemini apps in addition to training.Amazonbotalso powers Amazon products.GoogleOtheris, according to Google, a generic crawler with no effect on Google Search. - An obsolete token:
anthropic-aino longer appears in Anthropic's current documentation;ClaudeBot, which is also on the list, is the one that counts.
In short, checking the box blocks training for most major AI companies while letting the search crawlers of ChatGPT, Claude and Perplexity through. In exchange, you give up DuckAssist and grounding in Gemini, with no way to make an exception.
The target configuration, for comparison
Since the file cannot be edited, the following block is not meant to be pasted: it describes the intended effect, to help you read the file Squarespace serves.
# Blocked: training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
# Allowed: these crawlers are not named and follow the * group
# OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User,
# PerplexityBot, Perplexity-User
User-agent: *
Disallow: /config
Disallow: /api/
Reading the file Squarespace serves
One detail deserves attention. In the reference file at squarespace.com/robots.txt (checked in November 2026), and on other Squarespace sites we looked at, the 26 tokens from the list all appear at the top of the file, but in the same group as User-agent: *, that is, with only the general rules (Disallow: /config, Disallow: /api/, and so on):
User-agent: GPTBot
User-agent: Meta-ExternalAgent
User-agent: AdsBot-Google
User-agent: *
Disallow: /config
Disallow: /search
This is a simplified excerpt. Under RFC 9309, consecutive User-agent lines form a single group, so in this case GPTBot is not blocked: it gets exactly the same rules as every other crawler and can fetch every public page. We don't know the state of the checkbox on those sites, and Squarespace doesn't publish the exact file produced when it is checked. The check to make is therefore simple: when the setting is on, the User-agent: GPTBot line should belong to a group whose rule is Disallow: /, not to the User-agent: * group.
Squarespace-specific limits and pitfalls
- All or nothing. You can't keep
DuckAssistBotorGoogle-Extendedwhile blockingGPTBot, nor addMistralAI-Training. - Not retroactive. Squarespace points out that checking the box doesn't remove content that has already been collected from training data.
- A request, not a barrier.
robots.txtis a request that well-behaved crawlers respect; others can ignore it. Squarespace says making pages private is the only way to be sure a crawler never sees them. - Not the same as hiding a page. To remove a page from search results, Squarespace offers the "Hide page from search results" option in the SEO tab of the page settings (all plans, but not for the homepage or individual collection items), which adds a
noindextag and removes the page from the sitemap; it does not affectrobots.txt.
Check the result
Open https://www.example.com/robots.txt after saving the setting. Find the User-agent: GPTBot line and look at which rule follows its group. Then check a public page for GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot with the IABIG robots.txt tester: the first two should be disallowed, the other two allowed. If GPTBot is still allowed while the box is checked, report it to Squarespace support and include the contents of the file.
Official sources
- Squarespace, Request that AI models exclude your site: https://support.squarespace.com/hc/en-us/articles/360022347072-Request-that-AI-models-exclude-your-site
- Squarespace, Hiding your site from search results: https://support.squarespace.com/hc/en-us/articles/21829426828045-Hiding-your-site-from-search-results
- Squarespace, Understanding Google SEO emails and console errors: https://support.squarespace.com/hc/en-us/articles/206543207-Understanding-Google-SEO-emails-and-console-errors
- DuckDuckGo, DuckAssistBot: https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot
- Google, Google's common crawlers: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- Anthropic, Does Anthropic crawl data from the web: https://support.claude.com/en/articles/8896518
- RFC 9309, Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309
Free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
AI crawler blocklists
Ready-to-use files, refreshed hourly, to block AI training (or all AI crawlers): robots.txt, nginx, Apache, Cloudflare and the official IP ranges published by AI companies.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
Related guides
- How to Block AI Crawlers on IIS - robots.txt, URL Rewrite, Request Filtering and IP RestrictionsOn IIS, publish robots.txt, deny AI training crawlers with a URL Rewrite rule in web.config and learn the limits of request filtering and IP restrictions.
- How to Block AI Crawlers in nginx - User-Agent Rules, Official IP Ranges and Rate LimitsA tested nginx configuration that denies AI training crawlers, lets genuine AI search crawlers through, rate-limits them and writes them to a dedicated log.
- How to Block AI Crawlers in Apache 2.4 - robots.txt, .htaccess, Require not env and IP RangesDeny AI training crawlers in Apache 2.4 with .htaccess, mod_setenvif or mod_rewrite, and only accept AI search crawlers coming from their official IP ranges.
- Block or Allow AI Crawlers on WordPress - Virtual robots.txt, Physical File and PHP FilterWhere to edit robots.txt on WordPress to block GPTBot or ClaudeBot yet stay visible in ChatGPT Search, Claude and Perplexity, by file, PHP filter or plugin.