IABIGFR · Français

Platform guides · Updated 2026-10-25 · Markdown version

AI Crawlers on Next.js and Vercel - robots.ts, X-Robots-Tag and the Vercel Firewall

Configure robots.txt with app/robots.ts in Next.js, set an X-Robots-Tag header and tune the Vercel Firewall without blocking ChatGPT Search or Perplexity.

On a Next.js site hosted on Vercel, three layers can decide what happens to an AI crawler: the robots.txt file generated by the framework, the HTTP headers sent with each page, and the Vercel Firewall, which can reject a request before it ever reaches your application. They do not overlap. robots.txt states a preference that well-behaved crawlers honor, the firewall actually blocks, and an overly broad firewall setting can wipe a site out of ChatGPT Search or Perplexity even though robots.txt has nothing to do with it. This guide walks through each layer with the Next.js App Router, offers a configuration that refuses model training while letting search crawlers through, and explains how to verify and roll back every change.

Pick a single source for robots.txt

Next.js supports three ways to serve /robots.txt:

  • a static app/robots.txt file at the root of the app directory;
  • an app/robots.ts (or .js) file that returns an object describing the rules;
  • a public/robots.txt file, served as is like anything else in the public folder.

Keep only one. Two competing sources make the result hard to predict and, above all, hard to review six months later. A static file is enough if the rules never change; robots.ts becomes useful as soon as the content depends on the environment, for example to disallow everything on a staging domain.

The app/robots.ts file

The exported function returns an object of type MetadataRoute.Robots. Its rules property accepts a single object or an array of objects; within each object, userAgent, allow and disallow accept a string or an array of strings. The documentation also lists crawlDelay, sitemap, host and, since Next.js 16.3, an other field for non-standard directives.

// app/robots.ts
import type { MetadataRoute } from 'next'

// Areas no crawler should crawl (adjust to your site)
const EXCLUSIONS = ['/admin/', '/cart/']

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        // 1. Model training: disallowed (editorial choice)
        userAgent: [
          'GPTBot',
          'ClaudeBot',
          'CCBot',
          'meta-externalagent',
          'MistralAI-Training',
          'Bytespider',
          'Applebot-Extended',
          'Google-Extended',
        ],
        disallow: '/',
      },
      {
        // 2. Search and user-triggered fetching: allowed.
        //    These crawlers only apply this group, so the exclusions are repeated here.
        userAgent: [
          'OAI-SearchBot',
          'ChatGPT-User',
          'Claude-SearchBot',
          'Claude-User',
          'PerplexityBot',
          'Perplexity-User',
        ],
        allow: '/',
        disallow: EXCLUSIONS,
      },
      {
        // 3. All other crawlers (Googlebot, Bingbot, Applebot...)
        userAgent: '*',
        disallow: EXCLUSIONS,
      },
    ],
    sitemap: 'https://www.example.com/sitemap.xml',
  }
}

With Next.js 16.4.3, which we used to produce the output below, the user agents of a single object are merged into one group (excerpt):

User-Agent: GPTBot
User-Agent: ClaudeBot
User-Agent: CCBot
...
Disallow: /

User-Agent: OAI-SearchBot
User-Agent: Claude-SearchBot
...
Allow: /
Disallow: /admin/
Disallow: /cart/

User-Agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://www.example.com/sitemap.xml

The example in the official documentation shows a separate group for each user agent. Both forms are equivalent under RFC 9309, so judge the output by its content, not its exact layout.

Two further points from the documentation: robots.ts is a special route handler that is cached by default unless it uses a request-time API, and the file must sit at the root of app, not in a subfolder. What each token does and the usual syntax mistakes are covered in our guide on blocking AI training in robots.txt while staying visible in AI search.

The X-Robots-Tag header

X-Robots-Tag carries the same directives as a <meta name="robots"> tag, but in an HTTP header, which makes it useful for PDFs and other non-HTML responses. In Next.js you set it with the headers() function in next.config:

// next.config.ts
import type { NextConfig } from 'next'

const nextConfig: NextConfig = {
  async headers() {
    return [
      {
        // All files under /documents/: do not index
        source: '/documents/:path*',
        headers: [{ key: 'X-Robots-Tag', value: 'noindex' }],
      },
    ]
  },
}

export default nextConfig

Keep three limitations in mind. Google documents this header, including the targeted form X-Robots-Tag: googlebot: noindex, but the official OpenAI, Anthropic and Perplexity pages we consulted for this guide mention neither X-Robots-Tag nor noindex; for those crawlers, robots.txt remains the only documented control. Next, a crawler only sees the header if it is allowed to fetch the page: a URL disallowed in robots.txt never reveals its noindex. Finally, Vercel already adds X-Robots-Tag: noindex to preview deployments unless they are assigned a custom domain, so do not be surprised to see it when testing a preview URL.

The Vercel Firewall

In the project dashboard, Firewall then Rules shows a Bot Management section with two managed rules, available on all plans and inactive by default:

Managed rule Available actions Effect
Bot Protection Log or Challenge JavaScript challenge for traffic that does not behave like a browser; bots verified by Vercel, such as Googlebot, are excluded from evaluation
AI Bots Ruleset Log or Deny Detects known AI bots, whether they are used for training, search or user-triggered requests; Vercel maintains the list

The deciding factor lies in how the second rule is defined: it also covers search and on-demand fetching. Setting it to Deny therefore blocks, according to Vercel's description, OAI-SearchBot, PerplexityBot or Claude-User just as it blocks GPTBot, and your site drops out of answers that cite their sources. The documentation does not let you pick a category within this rule. We also could not find the exact list of bots it recognizes.

To follow the approach recommended in this guide (refuse training, keep search), use a custom rule instead: Add Rule, a User Agent condition with the Matches expression operator, the Deny action, then Review Changes and Publish.

Custom rule "Deny AI training"
  If    User Agent    Matches expression   (?i)(GPTBot|ClaudeBot|CCBot|meta-externalagent|Bytespider)
  And   Request Path  Does not equal       /robots.txt
  Then  Deny

The second condition keeps robots.txt readable: a crawler that cannot read the file cannot discover your refusal in it either. Vercel documents its operators as case-insensitive; the (?i) prefix only makes that explicit. Test the rule with the Log action first: matching requests show up on the firewall overview page without being blocked.

If you want to keep the managed rule on Deny, Vercel notes that a custom rule with the Bypass action can let specific traffic through. An exception based on the User-Agent alone, however, also lets through anyone who spoofs that name. Vercel also states that Bot Protection does not work well behind a reverse proxy placed in front of the deployment, such as another CDN.

Outside Vercel: proxy.ts

On a Node.js server or in a container, the equivalent logic goes in proxy.ts, the name middleware has carried since Next.js 16. Without a matcher, it runs for every request, static files included.

// proxy.ts (at the project root)
import { NextResponse } from 'next/server'
import type { NextRequest } from 'next/server'

const DENIED = /GPTBot|ClaudeBot|CCBot|meta-externalagent|Bytespider/i

export function proxy(request: NextRequest) {
  const ua = request.headers.get('user-agent') ?? ''
  if (DENIED.test(ua)) {
    return new NextResponse('Access denied', { status: 403 })
  }
  return NextResponse.next()
}

export const config = {
  // robots.txt and internal assets bypass the proxy
  matcher: ['/((?!_next/static|_next/image|favicon.ico|robots.txt|sitemap.xml).*)'],
}

On Vercel, prefer the firewall rule: it applies before your application, whereas the proxy is code that runs on every request.

Testing

# robots.txt served with a 200, as plain text
curl -I https://www.example.com/robots.txt

# Simulate a training crawler: 403 expected once the Deny rule is published
curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)" https://www.example.com/

# Simulate a search crawler: 200 expected
curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" https://www.example.com/

# X-Robots-Tag header on a document
curl -sI https://www.example.com/documents/pricing.pdf | grep -i x-robots-tag

These tests are only indicative: curl imitates the crawler's name, not its IP address, and a rule that checks where traffic comes from will not treat it as the real crawler. The check that matters happens afterwards in the project's Firewall view, where each blocked or logged request is tied to the rule that handled it. To write robots.txt rules without token mistakes, you can start from the AI robots.txt generator.

Pitfalls to avoid

  • Confusing a refusal with a block. robots.txt is enough for crawlers that honor it, which OpenAI, Anthropic and Perplexity state is the case for their crawlers. The firewall is only a complement, against those that ignore it.
  • Assuming a User-Agent filter stops impostors. It stops those that honestly announce their name. A scraper posing as Chrome gets through, and an overly broad filter can turn away a genuine search crawler.
  • Blocking Google-Extended in the firewall. It is a robots.txt token, not a crawler: no request ever carries that name. Blocking Googlebot instead would remove you from Google Search.
  • Forgetting subdomains and staging domains. Each one serves its own robots.txt.

Rolling back

For robots.txt, restore the previous version of the file and redeploy; crawlers re-read the file within roughly 24 hours. For the firewall, set AI Bots Ruleset back to Allow or Bot Protection back to Off, disable or delete the custom rule, then Review Changes and Publish. For X-Robots-Tag and proxy.ts, remove the relevant block and redeploy.

Official sources

Free tools

Related guides