# AI Crawlers on Netlify - robots.txt, the _headers File and an Edge Function Filter

> Publish robots.txt on Netlify, add X-Robots-Tag with _headers and filter AI crawlers by User-Agent with an Edge Function or the User Agent Blocker extension.

Source: https://iabig.com/guides/platforms/netlify-ai-crawlers/ — Guides CC BY 4.0, extraits de code MIT

Netlify serves static and hybrid sites from its own network, with no web server to configure: everything goes through files placed in the publish directory, through `netlify.toml` and, when needed, through code that runs at the edge of the network. For AI crawlers, that gives you three levers: the robots.txt file, which states your preferences; HTTP headers such as `X-Robots-Tag`; and actual request filtering, either with an Edge Function you write yourself or with the **User Agent Blocker** extension provided by Netlify. This guide covers each one, with a configuration that refuses model training while letting the search crawlers of ChatGPT, Claude and Perplexity through.

## Publishing robots.txt

On Netlify, robots.txt is a static file like any other: it must sit at the root of the publish directory. With a static site generator or a framework, put the file wherever the tool copies static files unchanged, often a `public/` or `static/` folder, then check after the build that it really ends up at the root of the deployed directory.

```text
# public/robots.txt - copied as is to the root of the published site

# 1. Model training: disallowed (editorial choice)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Bytespider
User-agent: Applebot-Extended
User-agent: Google-Extended
Disallow: /

# 2. Search and user-triggered fetching: allowed
#    These crawlers only apply this group, so the exclusions are repeated here.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /admin/
Allow: /

# 3. All other crawlers
User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml
```

Netlify's guide on blocking AI crawlers points out that not every AI product honors robots.txt. That is true, but the major vendors state that their crawlers do: start with this file, and only move on to filtering for the crawlers that ignore it.

## Adding X-Robots-Tag with _headers

The `_headers` file, also placed in the publish directory, maps paths to headers. Each path is followed by its indented headers, and lines starting with `#` are comments.

```text
# _headers (at the root of the publish directory)

# PDF documents: do not index
/documents/*
  X-Robots-Tag: noindex

# Internal search results pages
/search/*
  X-Robots-Tag: noindex, nofollow
```

The same rule can be written in `netlify.toml`:

```toml
[[headers]]
  for = "/documents/*"
  [headers.values]
    X-Robots-Tag = "noindex"
```

Two limitations documented by Netlify matter here. Custom headers only apply to files served from Netlify's storage: a page rendered server-side by a function or an Edge Function, or content served through a proxy, does not receive them, and it is then up to your code to send the header. In addition, headers declared in `_headers` or `netlify.toml` apply to every build and cannot be limited to a single deploy context.

As far as AI crawlers go, keep expectations in check: Google documents `X-Robots-Tag`, but the OpenAI, Anthropic and Perplexity pages we consulted for this guide do not mention it. For those crawlers, robots.txt remains the only documented control. And a crawler that robots.txt keeps away from a URL will never see that URL's header.

## The User Agent Blocker extension

Netlify offers a first-party extension that blocks requests from a predefined list of AI bots and SEO or search bots, itself built on an Edge Function. According to the documentation, setup works like this:

1. In the project dashboard, select **Extensions** in the left sidebar, search for **User Agent Blocker**, then click **Install** (the extension becomes available to the whole team).
2. In the relevant project, under **Extensions**, open **User Agent Blocker**.
3. Under **Block User Agents**, check the categories to block.
4. Click **Redeploy site**, confirm with **Save**, then **Enable analytics and deploy site**.

Netlify notes that the list is not exhaustive and that the extension can only be enabled on one project at a time. The documentation gives neither the names of the bots on the list, nor the HTTP status code returned, nor the plans it is available on: check what each box blocks before enabling it. Checking "all AI bots", as the page suggests, risks also shutting out the search crawlers that cite you.

## Writing your own Edge Function

To decide crawler by crawler, a few lines of Edge Function are enough. Files placed in `netlify/edge-functions/` are detected automatically, and a function that returns `undefined` lets the request continue on its way.

```ts
// netlify/edge-functions/block-ai-crawlers.ts
import type { Config, Context } from "@netlify/edge-functions";

// Denied crawlers: here, the training crawlers. Case-insensitive comparison.
const DENIED_CRAWLERS = [
  "GPTBot",
  "ClaudeBot",
  "CCBot",
  "meta-externalagent",
  "Bytespider",
];

export default async (request: Request, context: Context) => {
  const ua = (request.headers.get("user-agent") ?? "").toLowerCase();
  const crawler = DENIED_CRAWLERS.find((name) => ua.includes(name.toLowerCase()));

  if (crawler) {
    // Visible in the function logs
    console.log(`denied crawler: ${crawler} from ${context.ip}`);
    return new Response("Access denied to AI training crawlers.", {
      status: 403,
      headers: { "content-type": "text/plain; charset=utf-8" },
    });
  }
  // Nothing to do: the request continues normally.
  return;
};

export const config: Config = {
  path: "/*",
  // robots.txt must stay readable by every crawler
  excludedPath: ["/robots.txt"],
};
```

We checked this function's logic outside Netlify, with Node.js: a GPTBot or `ccbot` User-Agent gets a 403, while `OAI-SearchBot` and a browser go through. Excluding `/robots.txt` is essential, because a crawler that cannot read the file cannot discover your refusal in it. Netlify's documentation uses `URLPattern` patterns for `path` and `excludedPath`, and they must start with `/`.

## Rate limiting and IP filtering

Netlify also offers rate limiting rules. Rules written in code, for example in an Edge Function's `config` object with `windowLimit`, `windowSize` and `aggregateBy`, are available on all plans but can only target paths; over-limit requests get a 429 unless you set a rewrite. Rules based on a header such as `user-agent` are reserved for the Enterprise plan with High-Performance Edge. **Firewall Traffic Rules** block or allow by IP address or CIDR range, country or subregion, and a blocked visitor sees a Netlify-branded 404 page. On Free, Starter and Personal plans, a ruleset holds only 2 rules of 3 addresses each, far too few to mirror a vendor's published ranges: they are fine for shutting out an abusive range, not for recognizing a crawler.

## Testing

```bash
# robots.txt published correctly, with a 200
curl -I https://www.example.com/robots.txt

# Training crawler: 403 expected with the Edge Function
curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://www.example.com/

# robots.txt stays accessible to the same crawler: 200 expected
curl -s -o /dev/null -w '%{http_code}\n' -A "GPTBot/1.4" https://www.example.com/robots.txt

# Search crawler: 200 expected
curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://www.example.com/

# Header set by _headers
curl -sI https://www.example.com/documents/pricing.pdf | grep -i x-robots-tag
```

These tests are only indicative: they reproduce the crawler's name, not its IP address. The function's `console.log` messages appear in the project's Edge Functions logs, which lets you confirm that real requests are being denied. To check, crawler by crawler, which URLs your file allows, run it through the [IABIG robots.txt tester](/tools/robots-txt-tester/).

## Pitfalls to avoid

- **Assuming filtering stops impostors.** An Edge Function based on the User-Agent only sees what the client claims. A scraper posing as a browser gets through; to go further, you would need to compare `context.ip` with the ranges each vendor publishes.
- **Blocking search crawlers as a side effect.** A substring as short as `bot` denies everyone, Googlebot included.
- **Blocking `Google-Extended` or `Applebot-Extended` in a function.** These are robots.txt tokens, not crawlers: no request carries those names.
- **Forgetting server-rendered responses.** `_headers` does not apply to them.

## Rolling back

Delete or rename the Edge Function file, remove the relevant block from `_headers` or `netlify.toml`, or disable the extension, then redeploy. Netlify also lets you republish an earlier deploy from the deploys list. For robots.txt, allow up to 24 hours for crawlers to re-read the new version (RFC 9309 asks them not to cache it longer).

## Official sources

- Netlify, Block AI crawlers and bots: <https://docs.netlify.com/build/build-with-ai/block-ai-crawlers>
- Netlify, Blocking AI bots and controlling crawlers: <https://developers.netlify.com/guides/blocking-ai-bots-and-controlling-crawlers/>
- Netlify, Custom headers: <https://docs.netlify.com/manage/routing/headers/>
- Netlify, Edge Functions API: <https://docs.netlify.com/build/edge-functions/api/>
- Netlify, Edge Functions declarations: <https://docs.netlify.com/build/edge-functions/declarations/>
- Netlify, Rate limiting: <https://docs.netlify.com/manage/security/secure-access-to-sites/rate-limiting/>
- Netlify, Traffic rules: <https://docs.netlify.com/security/secure-access-to-sites/traffic-rules/>
- Google, Robots meta tag and X-Robots-Tag: <https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag>
- RFC 9309, Robots Exclusion Protocol: <https://www.rfc-editor.org/rfc/rfc9309>
- OpenAI, Overview of OpenAI Crawlers: <https://developers.openai.com/api/docs/bots>
- Anthropic, Does Anthropic crawl data from the web: <https://support.claude.com/en/articles/8896518>
- Perplexity, Perplexity Crawlers: <https://docs.perplexity.ai/docs/resources/perplexity-crawlers>
