IABIGFR · Français

Platform guides · Updated 2026-11-06 · Markdown version

AI Crawlers on Netlify - robots.txt, the _headers File and an Edge Function Filter

Publish robots.txt on Netlify, add X-Robots-Tag with _headers and filter AI crawlers by User-Agent with an Edge Function or the User Agent Blocker extension.

Netlify serves static and hybrid sites from its own network, with no web server to configure: everything goes through files placed in the publish directory, through netlify.toml and, when needed, through code that runs at the edge of the network. For AI crawlers, that gives you three levers: the robots.txt file, which states your preferences; HTTP headers such as X-Robots-Tag; and actual request filtering, either with an Edge Function you write yourself or with the User Agent Blocker extension provided by Netlify. This guide covers each one, with a configuration that refuses model training while letting the search crawlers of ChatGPT, Claude and Perplexity through.

Publishing robots.txt

On Netlify, robots.txt is a static file like any other: it must sit at the root of the publish directory. With a static site generator or a framework, put the file wherever the tool copies static files unchanged, often a public/ or static/ folder, then check after the build that it really ends up at the root of the deployed directory.

# public/robots.txt - copied as is to the root of the published site

# 1. Model training: disallowed (editorial choice)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Bytespider
User-agent: Applebot-Extended
User-agent: Google-Extended
Disallow: /

# 2. Search and user-triggered fetching: allowed
#    These crawlers only apply this group, so the exclusions are repeated here.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /admin/
Allow: /

# 3. All other crawlers
User-agent: *
Disallow: /admin/

Sitemap: https://www.example.com/sitemap.xml

Netlify's guide on blocking AI crawlers points out that not every AI product honors robots.txt. That is true, but the major vendors state that their crawlers do: start with this file, and only move on to filtering for the crawlers that ignore it.

Adding X-Robots-Tag with _headers

The _headers file, also placed in the publish directory, maps paths to headers. Each path is followed by its indented headers, and lines starting with # are comments.

# _headers (at the root of the publish directory)

# PDF documents: do not index
/documents/*
  X-Robots-Tag: noindex

# Internal search results pages
/search/*
  X-Robots-Tag: noindex, nofollow

The same rule can be written in netlify.toml:

[[headers]]
  for = "/documents/*"
  [headers.values]
    X-Robots-Tag = "noindex"

Two limitations documented by Netlify matter here. Custom headers only apply to files served from Netlify's storage: a page rendered server-side by a function or an Edge Function, or content served through a proxy, does not receive them, and it is then up to your code to send the header. In addition, headers declared in _headers or netlify.toml apply to every build and cannot be limited to a single deploy context.

As far as AI crawlers go, keep expectations in check: Google documents X-Robots-Tag, but the OpenAI, Anthropic and Perplexity pages we consulted for this guide do not mention it. For those crawlers, robots.txt remains the only documented control. And a crawler that robots.txt keeps away from a URL will never see that URL's header.

The User Agent Blocker extension

Netlify offers a first-party extension that blocks requests from a predefined list of AI bots and SEO or search bots, itself built on an Edge Function. According to the documentation, setup works like this:

  1. In the project dashboard, select Extensions in the left sidebar, search for User Agent Blocker, then click Install (the extension becomes available to the whole team).
  2. In the relevant project, under Extensions, open User Agent Blocker.
  3. Under Block User Agents, check the categories to block.
  4. Click Redeploy site, confirm with Save, then Enable analytics and deploy site.

Netlify notes that the list is not exhaustive and that the extension can only be enabled on one project at a time. The documentation gives neither the names of the bots on the list, nor the HTTP status code returned, nor the plans it is available on: check what each box blocks before enabling it. Checking "all AI bots", as the page suggests, risks also shutting out the search crawlers that cite you.

Writing your own Edge Function

To decide crawler by crawler, a few lines of Edge Function are enough. Files placed in netlify/edge-functions/ are detected automatically, and a function that returns undefined lets the request continue on its way.

// netlify/edge-functions/block-ai-crawlers.ts
import type { Config, Context } from "@netlify/edge-functions";

// Denied crawlers: here, the training crawlers. Case-insensitive comparison.
const DENIED_CRAWLERS = [
  "GPTBot",
  "ClaudeBot",
  "CCBot",
  "meta-externalagent",
  "Bytespider",
];

export default async (request: Request, context: Context) => {
  const ua = (request.headers.get("user-agent") ?? "").toLowerCase();
  const crawler = DENIED_CRAWLERS.find((name) => ua.includes(name.toLowerCase()));

  if (crawler) {
    // Visible in the function logs
    console.log(`denied crawler: ${crawler} from ${context.ip}`);
    return new Response("Access denied to AI training crawlers.", {
      status: 403,
      headers: { "content-type": "text/plain; charset=utf-8" },
    });
  }
  // Nothing to do: the request continues normally.
  return;
};

export const config: Config = {
  path: "/*",
  // robots.txt must stay readable by every crawler
  excludedPath: ["/robots.txt"],
};

We checked this function's logic outside Netlify, with Node.js: a GPTBot or ccbot User-Agent gets a 403, while OAI-SearchBot and a browser go through. Excluding /robots.txt is essential, because a crawler that cannot read the file cannot discover your refusal in it. Netlify's documentation uses URLPattern patterns for path and excludedPath, and they must start with /.

Rate limiting and IP filtering

Netlify also offers rate limiting rules. Rules written in code, for example in an Edge Function's config object with windowLimit, windowSize and aggregateBy, are available on all plans but can only target paths; over-limit requests get a 429 unless you set a rewrite. Rules based on a header such as user-agent are reserved for the Enterprise plan with High-Performance Edge. Firewall Traffic Rules block or allow by IP address or CIDR range, country or subregion, and a blocked visitor sees a Netlify-branded 404 page. On Free, Starter and Personal plans, a ruleset holds only 2 rules of 3 addresses each, far too few to mirror a vendor's published ranges: they are fine for shutting out an abusive range, not for recognizing a crawler.

Testing

# robots.txt published correctly, with a 200
curl -I https://www.example.com/robots.txt

# Training crawler: 403 expected with the Edge Function
curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://www.example.com/

# robots.txt stays accessible to the same crawler: 200 expected
curl -s -o /dev/null -w '%{http_code}\n' -A "GPTBot/1.4" https://www.example.com/robots.txt

# Search crawler: 200 expected
curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" https://www.example.com/

# Header set by _headers
curl -sI https://www.example.com/documents/pricing.pdf | grep -i x-robots-tag

These tests are only indicative: they reproduce the crawler's name, not its IP address. The function's console.log messages appear in the project's Edge Functions logs, which lets you confirm that real requests are being denied. To check, crawler by crawler, which URLs your file allows, run it through the IABIG robots.txt tester.

Pitfalls to avoid

  • Assuming filtering stops impostors. An Edge Function based on the User-Agent only sees what the client claims. A scraper posing as a browser gets through; to go further, you would need to compare context.ip with the ranges each vendor publishes.
  • Blocking search crawlers as a side effect. A substring as short as bot denies everyone, Googlebot included.
  • Blocking Google-Extended or Applebot-Extended in a function. These are robots.txt tokens, not crawlers: no request carries those names.
  • Forgetting server-rendered responses. _headers does not apply to them.

Rolling back

Delete or rename the Edge Function file, remove the relevant block from _headers or netlify.toml, or disable the extension, then redeploy. Netlify also lets you republish an earlier deploy from the deploys list. For robots.txt, allow up to 24 hours for crawlers to re-read the new version (RFC 9309 asks them not to cache it longer).

Official sources

Free tools

Related guides