# AI Crawlers on Next.js and Vercel - robots.ts, X-Robots-Tag and the Vercel Firewall

> Configure robots.txt with app/robots.ts in Next.js, set an X-Robots-Tag header and tune the Vercel Firewall without blocking ChatGPT Search or Perplexity.

Source: https://iabig.com/guides/platforms/nextjs-vercel-ai-crawlers/ — Guides CC BY 4.0, extraits de code MIT

On a Next.js site hosted on Vercel, three layers can decide what happens to an AI crawler: the robots.txt file generated by the framework, the HTTP headers sent with each page, and the Vercel Firewall, which can reject a request before it ever reaches your application. They do not overlap. robots.txt states a preference that well-behaved crawlers honor, the firewall actually blocks, and an overly broad firewall setting can wipe a site out of ChatGPT Search or Perplexity even though robots.txt has nothing to do with it. This guide walks through each layer with the Next.js App Router, offers a configuration that refuses model training while letting search crawlers through, and explains how to verify and roll back every change.

## Pick a single source for robots.txt

Next.js supports three ways to serve `/robots.txt`:

- a static `app/robots.txt` file at the root of the `app` directory;
- an `app/robots.ts` (or `.js`) file that returns an object describing the rules;
- a `public/robots.txt` file, served as is like anything else in the `public` folder.

Keep only one. Two competing sources make the result hard to predict and, above all, hard to review six months later. A static file is enough if the rules never change; `robots.ts` becomes useful as soon as the content depends on the environment, for example to disallow everything on a staging domain.

## The app/robots.ts file

The exported function returns an object of type `MetadataRoute.Robots`. Its `rules` property accepts a single object or an array of objects; within each object, `userAgent`, `allow` and `disallow` accept a string or an array of strings. The documentation also lists `crawlDelay`, `sitemap`, `host` and, since Next.js 16.3, an `other` field for non-standard directives.

```ts
// app/robots.ts
import type { MetadataRoute } from 'next'

// Areas no crawler should crawl (adjust to your site)
const EXCLUSIONS = ['/admin/', '/cart/']

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        // 1. Model training: disallowed (editorial choice)
        userAgent: [
          'GPTBot',
          'ClaudeBot',
          'CCBot',
          'meta-externalagent',
          'MistralAI-Training',
          'Bytespider',
          'Applebot-Extended',
          'Google-Extended',
        ],
        disallow: '/',
      },
      {
        // 2. Search and user-triggered fetching: allowed.
        //    These crawlers only apply this group, so the exclusions are repeated here.
        userAgent: [
          'OAI-SearchBot',
          'ChatGPT-User',
          'Claude-SearchBot',
          'Claude-User',
          'PerplexityBot',
          'Perplexity-User',
        ],
        allow: '/',
        disallow: EXCLUSIONS,
      },
      {
        // 3. All other crawlers (Googlebot, Bingbot, Applebot...)
        userAgent: '*',
        disallow: EXCLUSIONS,
      },
    ],
    sitemap: 'https://www.example.com/sitemap.xml',
  }
}
```

With Next.js 16.4.3, which we used to produce the output below, the user agents of a single object are merged into one group (excerpt):

```text
User-Agent: GPTBot
User-Agent: ClaudeBot
User-Agent: CCBot
...
Disallow: /

User-Agent: OAI-SearchBot
User-Agent: Claude-SearchBot
...
Allow: /
Disallow: /admin/
Disallow: /cart/

User-Agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml
```

The example in the official documentation shows a separate group for each user agent. Both forms are equivalent under RFC 9309, so judge the output by its content, not its exact layout.

Two further points from the documentation: `robots.ts` is a special route handler that is cached by default unless it uses a request-time API, and the file must sit at the root of `app`, not in a subfolder. What each token does and the usual syntax mistakes are covered in our guide on [blocking AI training in robots.txt while staying visible in AI search](/guides/ai-crawlers/block-ai-training-robots-txt/).

## The X-Robots-Tag header

`X-Robots-Tag` carries the same directives as a `<meta name="robots">` tag, but in an HTTP header, which makes it useful for PDFs and other non-HTML responses. In Next.js you set it with the `headers()` function in `next.config`:

```ts
// next.config.ts
import type { NextConfig } from 'next'

const nextConfig: NextConfig = {
  async headers() {
    return [
      {
        // All files under /documents/: do not index
        source: '/documents/:path*',
        headers: [{ key: 'X-Robots-Tag', value: 'noindex' }],
      },
    ]
  },
}

export default nextConfig
```

Keep three limitations in mind. Google documents this header, including the targeted form `X-Robots-Tag: googlebot: noindex`, but the official OpenAI, Anthropic and Perplexity pages we consulted for this guide mention neither `X-Robots-Tag` nor `noindex`; for those crawlers, robots.txt remains the only documented control. Next, a crawler only sees the header if it is allowed to fetch the page: a URL disallowed in robots.txt never reveals its `noindex`. Finally, Vercel already adds `X-Robots-Tag: noindex` to preview deployments unless they are assigned a custom domain, so do not be surprised to see it when testing a preview URL.

## The Vercel Firewall

In the project dashboard, **Firewall** then **Rules** shows a **Bot Management** section with two managed rules, available on all plans and inactive by default:

| Managed rule | Available actions | Effect |
|---|---|---|
| **Bot Protection** | Log or Challenge | JavaScript challenge for traffic that does not behave like a browser; bots verified by Vercel, such as Googlebot, are excluded from evaluation |
| **AI Bots Ruleset** | Log or Deny | Detects known AI bots, whether they are used for training, search or user-triggered requests; Vercel maintains the list |

The deciding factor lies in how the second rule is defined: it also covers search and on-demand fetching. Setting it to **Deny** therefore blocks, according to Vercel's description, `OAI-SearchBot`, `PerplexityBot` or `Claude-User` just as it blocks `GPTBot`, and your site drops out of answers that cite their sources. The documentation does not let you pick a category within this rule. We also could not find the exact list of bots it recognizes.

To follow the approach recommended in this guide (refuse training, keep search), use a custom rule instead: **Add Rule**, a **User Agent** condition with the **Matches expression** operator, the **Deny** action, then **Review Changes** and **Publish**.

```text
Custom rule "Deny AI training"
  If    User Agent    Matches expression   (?i)(GPTBot|ClaudeBot|CCBot|meta-externalagent|Bytespider)
  And   Request Path  Does not equal       /robots.txt
  Then  Deny
```

The second condition keeps `robots.txt` readable: a crawler that cannot read the file cannot discover your refusal in it either. Vercel documents its operators as case-insensitive; the `(?i)` prefix only makes that explicit. Test the rule with the **Log** action first: matching requests show up on the firewall overview page without being blocked.

If you want to keep the managed rule on **Deny**, Vercel notes that a custom rule with the **Bypass** action can let specific traffic through. An exception based on the User-Agent alone, however, also lets through anyone who spoofs that name. Vercel also states that Bot Protection does not work well behind a reverse proxy placed in front of the deployment, such as another CDN.

## Outside Vercel: proxy.ts

On a Node.js server or in a container, the equivalent logic goes in `proxy.ts`, the name middleware has carried since Next.js 16. Without a `matcher`, it runs for every request, static files included.

```ts
// proxy.ts (at the project root)
import { NextResponse } from 'next/server'
import type { NextRequest } from 'next/server'

const DENIED = /GPTBot|ClaudeBot|CCBot|meta-externalagent|Bytespider/i

export function proxy(request: NextRequest) {
  const ua = request.headers.get('user-agent') ?? ''
  if (DENIED.test(ua)) {
    return new NextResponse('Access denied', { status: 403 })
  }
  return NextResponse.next()
}

export const config = {
  // robots.txt and internal assets bypass the proxy
  matcher: ['/((?!_next/static|_next/image|favicon.ico|robots.txt|sitemap.xml).*)'],
}
```

On Vercel, prefer the firewall rule: it applies before your application, whereas the proxy is code that runs on every request.

## Testing

```bash
# robots.txt served with a 200, as plain text
curl -I https://www.example.com/robots.txt

# Simulate a training crawler: 403 expected once the Deny rule is published
curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)" https://www.example.com/

# Simulate a search crawler: 200 expected
curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" https://www.example.com/

# X-Robots-Tag header on a document
curl -sI https://www.example.com/documents/pricing.pdf | grep -i x-robots-tag
```

These tests are only indicative: curl imitates the crawler's name, not its IP address, and a rule that checks where traffic comes from will not treat it as the real crawler. The check that matters happens afterwards in the project's **Firewall** view, where each blocked or logged request is tied to the rule that handled it. To write robots.txt rules without token mistakes, you can start from the [AI robots.txt generator](/tools/robots-txt-generator/).

## Pitfalls to avoid

- **Confusing a refusal with a block.** robots.txt is enough for crawlers that honor it, which OpenAI, Anthropic and Perplexity state is the case for their crawlers. The firewall is only a complement, against those that ignore it.
- **Assuming a User-Agent filter stops impostors.** It stops those that honestly announce their name. A scraper posing as Chrome gets through, and an overly broad filter can turn away a genuine search crawler.
- **Blocking `Google-Extended` in the firewall.** It is a robots.txt token, not a crawler: no request ever carries that name. Blocking `Googlebot` instead would remove you from Google Search.
- **Forgetting subdomains and staging domains.** Each one serves its own `robots.txt`.

## Rolling back

For robots.txt, restore the previous version of the file and redeploy; crawlers re-read the file within roughly 24 hours. For the firewall, set **AI Bots Ruleset** back to **Allow** or **Bot Protection** back to **Off**, disable or delete the custom rule, then **Review Changes** and **Publish**. For `X-Robots-Tag` and `proxy.ts`, remove the relevant block and redeploy.

## Official sources

- Next.js, robots.txt (Metadata API): <https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots>
- Next.js, headers in next.config: <https://nextjs.org/docs/app/api-reference/config/next-config-js/headers>
- Next.js, proxy.js: <https://nextjs.org/docs/app/api-reference/file-conventions/proxy>
- Vercel, Bot Management: <https://vercel.com/docs/bot-management>
- Vercel, WAF Managed Rulesets: <https://vercel.com/docs/vercel-firewall/vercel-waf/managed-rulesets>
- Vercel, Rule Configuration Reference: <https://vercel.com/docs/vercel-firewall/vercel-waf/rule-configuration>
- Vercel, How to block bots from OpenAI GPTBot: <https://vercel.com/kb/guide/how-to-block-bots-openai-gptbot>
- Vercel, Are Vercel Preview Deployments indexed by search engines: <https://vercel.com/kb/guide/are-vercel-preview-deployment-indexed-by-search-engines>
- Google, Robots meta tag and X-Robots-Tag: <https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag>
- RFC 9309, Robots Exclusion Protocol: <https://www.rfc-editor.org/rfc/rfc9309>
- OpenAI, Overview of OpenAI Crawlers: <https://developers.openai.com/api/docs/bots>
- Anthropic, Does Anthropic crawl data from the web: <https://support.claude.com/en/articles/8896518>
- Perplexity, Perplexity Crawlers: <https://docs.perplexity.ai/docs/resources/perplexity-crawlers>
