IABIGFR · Français

Platform guides · Updated 2026-11-06 · Markdown version

Block or Allow AI Crawlers on Squarespace - What the Crawlers Setting Really Does

Squarespace doesn't let you edit robots.txt. What its AI crawler checkbox blocks, what it lets through, and how to check the file that is actually served.

Squarespace is the most locked-down of the major platforms on this point: the robots.txt file is shared by all sites and cannot be edited. The only lever on offer is a checkbox that adds a list of AI crawlers, chosen by Squarespace, to the file. This guide details exactly what that list covers and what it doesn't, why it partly fits the goal of "block training, stay visible in search", and how to check the file once the setting is applied.

No robots.txt editing

Squarespace's documentation is explicit: every Squarespace site uses the same robots.txt file, and users can neither access nor modify it. Squarespace justifies this as following SEO best practices. Among other things, the file excludes /config/ (the admin login page), /api/ (which Squarespace says blocks its analytics tracking cookie), /search, and URLs that duplicate content, such as lists filtered by author, tag or month. Squarespace points to squarespace.com/robots.txt as the reference copy of the full file.

Two settings affect this file, both in the same panel:

  • Block Search engine crawlers, which asks search engines not to index the site by adding a Disallow: / rule. It removes the site from search engines: don't check it if you want visibility, including in AI assistants.
  • Block known artificial intelligence crawlers, which adds an exclusion request for a list of AI crawlers.

There is no finer-grained setting. Squarespace states that you cannot ask AI crawlers to crawl only certain pages.

Where to find the setting

The path Squarespace gives is:

  1. Open the Settings panel.
  2. Click Crawlers.
  3. Check Block known artificial intelligence crawlers.

The box is unchecked by default. Squarespace explains this choice by its concern about reducing sites' traffic by excluding them from the answers and cited sources of AI assistants.

What the checkbox blocks exactly

According to Squarespace's help article (updated January 8, 2026), checking the box asks the following crawlers not to crawl the site:

AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended,
Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler,
DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image,
GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent,
MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot

Squarespace says this list may grow, and that you can suggest a missing crawler through a feature request.

Measured against the goal of this site, the list reads as follows:

  • The main training crawlers are on it: GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent (case doesn't matter in the crawler name), Google-Extended and Applebot-Extended. MistralAI-Training is not.
  • The AI assistants' search crawlers are not on it: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User remain subject to the general * group, so they are allowed on public pages.
  • Some blocked crawlers are also used to cite you. DuckDuckGo presents DuckAssistBot as the crawler behind its AI-assisted answers and states that its data is not used to train models. Google-Extended covers grounding of answers in the Gemini apps in addition to training. Amazonbot also powers Amazon products. GoogleOther is, according to Google, a generic crawler with no effect on Google Search.
  • An obsolete token: anthropic-ai no longer appears in Anthropic's current documentation; ClaudeBot, which is also on the list, is the one that counts.

In short, checking the box blocks training for most major AI companies while letting the search crawlers of ChatGPT, Claude and Perplexity through. In exchange, you give up DuckAssist and grounding in Gemini, with no way to make an exception.

The target configuration, for comparison

Since the file cannot be edited, the following block is not meant to be pasted: it describes the intended effect, to help you read the file Squarespace serves.

# Blocked: training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Allowed: these crawlers are not named and follow the * group
# OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User,
# PerplexityBot, Perplexity-User
User-agent: *
Disallow: /config
Disallow: /api/

Reading the file Squarespace serves

One detail deserves attention. In the reference file at squarespace.com/robots.txt (checked in November 2026), and on other Squarespace sites we looked at, the 26 tokens from the list all appear at the top of the file, but in the same group as User-agent: *, that is, with only the general rules (Disallow: /config, Disallow: /api/, and so on):

User-agent: GPTBot
User-agent: Meta-ExternalAgent
User-agent: AdsBot-Google
User-agent: *
Disallow: /config
Disallow: /search

This is a simplified excerpt. Under RFC 9309, consecutive User-agent lines form a single group, so in this case GPTBot is not blocked: it gets exactly the same rules as every other crawler and can fetch every public page. We don't know the state of the checkbox on those sites, and Squarespace doesn't publish the exact file produced when it is checked. The check to make is therefore simple: when the setting is on, the User-agent: GPTBot line should belong to a group whose rule is Disallow: /, not to the User-agent: * group.

Squarespace-specific limits and pitfalls

  • All or nothing. You can't keep DuckAssistBot or Google-Extended while blocking GPTBot, nor add MistralAI-Training.
  • Not retroactive. Squarespace points out that checking the box doesn't remove content that has already been collected from training data.
  • A request, not a barrier. robots.txt is a request that well-behaved crawlers respect; others can ignore it. Squarespace says making pages private is the only way to be sure a crawler never sees them.
  • Not the same as hiding a page. To remove a page from search results, Squarespace offers the "Hide page from search results" option in the SEO tab of the page settings (all plans, but not for the homepage or individual collection items), which adds a noindex tag and removes the page from the sitemap; it does not affect robots.txt.

Check the result

Open https://www.example.com/robots.txt after saving the setting. Find the User-agent: GPTBot line and look at which rule follows its group. Then check a public page for GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot with the IABIG robots.txt tester: the first two should be disallowed, the other two allowed. If GPTBot is still allowed while the box is checked, report it to Squarespace support and include the contents of the file.

Official sources

Free tools

Related guides