# Block or Allow AI Crawlers on Squarespace - What the Crawlers Setting Really Does

> Squarespace doesn't let you edit robots.txt. What its AI crawler checkbox blocks, what it lets through, and how to check the file that is actually served.

Source: https://iabig.com/guides/platforms/squarespace-ai-crawlers/ — Guides CC BY 4.0, extraits de code MIT

Squarespace is the most locked-down of the major platforms on this point: the `robots.txt` file is shared by all sites and cannot be edited. The only lever on offer is a checkbox that adds a list of AI crawlers, chosen by Squarespace, to the file. This guide details exactly what that list covers and what it doesn't, why it partly fits the goal of "block training, stay visible in search", and how to check the file once the setting is applied.

## No robots.txt editing

Squarespace's documentation is explicit: every Squarespace site uses the same `robots.txt` file, and users can neither access nor modify it. Squarespace justifies this as following SEO best practices. Among other things, the file excludes `/config/` (the admin login page), `/api/` (which Squarespace says blocks its analytics tracking cookie), `/search`, and URLs that duplicate content, such as lists filtered by author, tag or month. Squarespace points to `squarespace.com/robots.txt` as the reference copy of the full file.

Two settings affect this file, both in the same panel:

- **Block Search engine crawlers**, which asks search engines not to index the site by adding a `Disallow: /` rule. It removes the site from search engines: don't check it if you want visibility, including in AI assistants.
- **Block known artificial intelligence crawlers**, which adds an exclusion request for a list of AI crawlers.

There is no finer-grained setting. Squarespace states that you cannot ask AI crawlers to crawl only certain pages.

## Where to find the setting

The path Squarespace gives is:

1. Open the **Settings** panel.
2. Click **Crawlers**.
3. Check **Block known artificial intelligence crawlers**.

The box is unchecked by default. Squarespace explains this choice by its concern about reducing sites' traffic by excluding them from the answers and cited sources of AI assistants.

## What the checkbox blocks exactly

According to Squarespace's help article (updated January 8, 2026), checking the box asks the following crawlers not to crawl the site:

```text
AI2Bot, Ai2Bot-Dolma, aiHitBot, Amazonbot, anthropic-ai, Applebot-Extended,
Bytespider, CCBot, ClaudeBot, cohere-ai, cohere-training-data-crawler,
DuckAssistBot, FacebookBot, Google-Extended, GoogleOther, GoogleOther-Image,
GoogleOther-Video, GPTBot, img2dataset, Meta-ExternalAgent,
MyCentralAIScraperBot, omgili, omgilibot, Quora-Bot, TikTokSpider, YouBot
```

Squarespace says this list may grow, and that you can suggest a missing crawler through a feature request.

Measured against the goal of this site, the list reads as follows:

- **The main training crawlers are on it**: `GPTBot`, `ClaudeBot`, `CCBot`, `Meta-ExternalAgent` (case doesn't matter in the crawler name), `Google-Extended` and `Applebot-Extended`. `MistralAI-Training` is not.
- **The AI assistants' search crawlers are not on it**: `OAI-SearchBot`, `ChatGPT-User`, `Claude-SearchBot`, `Claude-User`, `PerplexityBot` and `Perplexity-User` remain subject to the general `*` group, so they are allowed on public pages.
- **Some blocked crawlers are also used to cite you.** DuckDuckGo presents `DuckAssistBot` as the crawler behind its AI-assisted answers and states that its data is not used to train models. `Google-Extended` covers grounding of answers in the Gemini apps in addition to training. `Amazonbot` also powers Amazon products. `GoogleOther` is, according to Google, a generic crawler with no effect on Google Search.
- **An obsolete token**: `anthropic-ai` no longer appears in Anthropic's current documentation; `ClaudeBot`, which is also on the list, is the one that counts.

In short, checking the box blocks training for most major AI companies while letting the search crawlers of ChatGPT, Claude and Perplexity through. In exchange, you give up DuckAssist and grounding in Gemini, with no way to make an exception.

## The target configuration, for comparison

Since the file cannot be edited, the following block is not meant to be pasted: it describes the intended effect, to help you read the file Squarespace serves.

```text
# Blocked: training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Allowed: these crawlers are not named and follow the * group
# OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User,
# PerplexityBot, Perplexity-User
User-agent: *
Disallow: /config
Disallow: /api/
```

## Reading the file Squarespace serves

One detail deserves attention. In the reference file at `squarespace.com/robots.txt` (checked in November 2026), and on other Squarespace sites we looked at, the 26 tokens from the list all appear at the top of the file, but in the same group as `User-agent: *`, that is, with only the general rules (`Disallow: /config`, `Disallow: /api/`, and so on):

```text
User-agent: GPTBot
User-agent: Meta-ExternalAgent
User-agent: AdsBot-Google
User-agent: *
Disallow: /config
Disallow: /search
```

This is a simplified excerpt. Under RFC 9309, consecutive `User-agent` lines form a single group, so in this case `GPTBot` is not blocked: it gets exactly the same rules as every other crawler and can fetch every public page. We don't know the state of the checkbox on those sites, and Squarespace doesn't publish the exact file produced when it is checked. The check to make is therefore simple: when the setting is on, the `User-agent: GPTBot` line should belong to a group whose rule is `Disallow: /`, not to the `User-agent: *` group.

## Squarespace-specific limits and pitfalls

- **All or nothing.** You can't keep `DuckAssistBot` or `Google-Extended` while blocking `GPTBot`, nor add `MistralAI-Training`.
- **Not retroactive.** Squarespace points out that checking the box doesn't remove content that has already been collected from training data.
- **A request, not a barrier.** `robots.txt` is a request that well-behaved crawlers respect; others can ignore it. Squarespace says making pages private is the only way to be sure a crawler never sees them.
- **Not the same as hiding a page.** To remove a page from search results, Squarespace offers the "Hide page from search results" option in the SEO tab of the page settings (all plans, but not for the homepage or individual collection items), which adds a `noindex` tag and removes the page from the sitemap; it does not affect `robots.txt`.

## Check the result

Open `https://www.example.com/robots.txt` after saving the setting. Find the `User-agent: GPTBot` line and look at which rule follows its group. Then check a public page for `GPTBot`, `ClaudeBot`, `OAI-SearchBot` and `PerplexityBot` with the [IABIG robots.txt tester](/tools/robots-txt-tester/): the first two should be disallowed, the other two allowed. If `GPTBot` is still allowed while the box is checked, report it to Squarespace support and include the contents of the file.

## Official sources

- Squarespace, Request that AI models exclude your site: <https://support.squarespace.com/hc/en-us/articles/360022347072-Request-that-AI-models-exclude-your-site>
- Squarespace, Hiding your site from search results: <https://support.squarespace.com/hc/en-us/articles/21829426828045-Hiding-your-site-from-search-results>
- Squarespace, Understanding Google SEO emails and console errors: <https://support.squarespace.com/hc/en-us/articles/206543207-Understanding-Google-SEO-emails-and-console-errors>
- DuckDuckGo, DuckAssistBot: <https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot>
- Google, Google's common crawlers: <https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers>
- Anthropic, Does Anthropic crawl data from the web: <https://support.claude.com/en/articles/8896518>
- RFC 9309, Robots Exclusion Protocol: <https://www.rfc-editor.org/rfc/rfc9309>
