AI Crawlers on Next.js and Vercel - robots.ts, X-Robots-Tag and the Vercel Firewall
Configure robots.txt with app/robots.ts in Next.js, set an X-Robots-Tag header and tune the Vercel Firewall without blocking ChatGPT Search or Perplexity.
On a Next.js site hosted on Vercel, three layers can decide what happens to an AI crawler: the robots.txt file generated by the framework, the HTTP headers sent with each page, and the Vercel Firewall, which can reject a request before it ever reaches your application. They do not overlap. robots.txt states a preference that well-behaved crawlers honor, the firewall actually blocks, and an overly broad firewall setting can wipe a site out of ChatGPT Search or Perplexity even though robots.txt has nothing to do with it. This guide walks through each layer with the Next.js App Router, offers a configuration that refuses model training while letting search crawlers through, and explains how to verify and roll back every change.
Pick a single source for robots.txt
Next.js supports three ways to serve /robots.txt:
- a static
app/robots.txtfile at the root of theappdirectory; - an
app/robots.ts(or.js) file that returns an object describing the rules; - a
public/robots.txtfile, served as is like anything else in thepublicfolder.
Keep only one. Two competing sources make the result hard to predict and, above all, hard to review six months later. A static file is enough if the rules never change; robots.ts becomes useful as soon as the content depends on the environment, for example to disallow everything on a staging domain.
The app/robots.ts file
The exported function returns an object of type MetadataRoute.Robots. Its rules property accepts a single object or an array of objects; within each object, userAgent, allow and disallow accept a string or an array of strings. The documentation also lists crawlDelay, sitemap, host and, since Next.js 16.3, an other field for non-standard directives.
// app/robots.ts
import type { MetadataRoute } from 'next'
// Areas no crawler should crawl (adjust to your site)
const EXCLUSIONS = ['/admin/', '/cart/']
export default function robots(): MetadataRoute.Robots {
return {
rules: [
{
// 1. Model training: disallowed (editorial choice)
userAgent: [
'GPTBot',
'ClaudeBot',
'CCBot',
'meta-externalagent',
'MistralAI-Training',
'Bytespider',
'Applebot-Extended',
'Google-Extended',
],
disallow: '/',
},
{
// 2. Search and user-triggered fetching: allowed.
// These crawlers only apply this group, so the exclusions are repeated here.
userAgent: [
'OAI-SearchBot',
'ChatGPT-User',
'Claude-SearchBot',
'Claude-User',
'PerplexityBot',
'Perplexity-User',
],
allow: '/',
disallow: EXCLUSIONS,
},
{
// 3. All other crawlers (Googlebot, Bingbot, Applebot...)
userAgent: '*',
disallow: EXCLUSIONS,
},
],
sitemap: 'https://www.example.com/sitemap.xml',
}
}
With Next.js 16.4.3, which we used to produce the output below, the user agents of a single object are merged into one group (excerpt):
User-Agent: GPTBot
User-Agent: ClaudeBot
User-Agent: CCBot
...
Disallow: /
User-Agent: OAI-SearchBot
User-Agent: Claude-SearchBot
...
Allow: /
Disallow: /admin/
Disallow: /cart/
User-Agent: *
Disallow: /admin/
Disallow: /cart/
Sitemap: https://www.example.com/sitemap.xml
The example in the official documentation shows a separate group for each user agent. Both forms are equivalent under RFC 9309, so judge the output by its content, not its exact layout.
Two further points from the documentation: robots.ts is a special route handler that is cached by default unless it uses a request-time API, and the file must sit at the root of app, not in a subfolder. What each token does and the usual syntax mistakes are covered in our guide on blocking AI training in robots.txt while staying visible in AI search.
The X-Robots-Tag header
X-Robots-Tag carries the same directives as a <meta name="robots"> tag, but in an HTTP header, which makes it useful for PDFs and other non-HTML responses. In Next.js you set it with the headers() function in next.config:
// next.config.ts
import type { NextConfig } from 'next'
const nextConfig: NextConfig = {
async headers() {
return [
{
// All files under /documents/: do not index
source: '/documents/:path*',
headers: [{ key: 'X-Robots-Tag', value: 'noindex' }],
},
]
},
}
export default nextConfig
Keep three limitations in mind. Google documents this header, including the targeted form X-Robots-Tag: googlebot: noindex, but the official OpenAI, Anthropic and Perplexity pages we consulted for this guide mention neither X-Robots-Tag nor noindex; for those crawlers, robots.txt remains the only documented control. Next, a crawler only sees the header if it is allowed to fetch the page: a URL disallowed in robots.txt never reveals its noindex. Finally, Vercel already adds X-Robots-Tag: noindex to preview deployments unless they are assigned a custom domain, so do not be surprised to see it when testing a preview URL.
The Vercel Firewall
In the project dashboard, Firewall then Rules shows a Bot Management section with two managed rules, available on all plans and inactive by default:
| Managed rule | Available actions | Effect |
|---|---|---|
| Bot Protection | Log or Challenge | JavaScript challenge for traffic that does not behave like a browser; bots verified by Vercel, such as Googlebot, are excluded from evaluation |
| AI Bots Ruleset | Log or Deny | Detects known AI bots, whether they are used for training, search or user-triggered requests; Vercel maintains the list |
The deciding factor lies in how the second rule is defined: it also covers search and on-demand fetching. Setting it to Deny therefore blocks, according to Vercel's description, OAI-SearchBot, PerplexityBot or Claude-User just as it blocks GPTBot, and your site drops out of answers that cite their sources. The documentation does not let you pick a category within this rule. We also could not find the exact list of bots it recognizes.
To follow the approach recommended in this guide (refuse training, keep search), use a custom rule instead: Add Rule, a User Agent condition with the Matches expression operator, the Deny action, then Review Changes and Publish.
Custom rule "Deny AI training"
If User Agent Matches expression (?i)(GPTBot|ClaudeBot|CCBot|meta-externalagent|Bytespider)
And Request Path Does not equal /robots.txt
Then Deny
The second condition keeps robots.txt readable: a crawler that cannot read the file cannot discover your refusal in it either. Vercel documents its operators as case-insensitive; the (?i) prefix only makes that explicit. Test the rule with the Log action first: matching requests show up on the firewall overview page without being blocked.
If you want to keep the managed rule on Deny, Vercel notes that a custom rule with the Bypass action can let specific traffic through. An exception based on the User-Agent alone, however, also lets through anyone who spoofs that name. Vercel also states that Bot Protection does not work well behind a reverse proxy placed in front of the deployment, such as another CDN.
Outside Vercel: proxy.ts
On a Node.js server or in a container, the equivalent logic goes in proxy.ts, the name middleware has carried since Next.js 16. Without a matcher, it runs for every request, static files included.
// proxy.ts (at the project root)
import { NextResponse } from 'next/server'
import type { NextRequest } from 'next/server'
const DENIED = /GPTBot|ClaudeBot|CCBot|meta-externalagent|Bytespider/i
export function proxy(request: NextRequest) {
const ua = request.headers.get('user-agent') ?? ''
if (DENIED.test(ua)) {
return new NextResponse('Access denied', { status: 403 })
}
return NextResponse.next()
}
export const config = {
// robots.txt and internal assets bypass the proxy
matcher: ['/((?!_next/static|_next/image|favicon.ico|robots.txt|sitemap.xml).*)'],
}
On Vercel, prefer the firewall rule: it applies before your application, whereas the proxy is code that runs on every request.
Testing
# robots.txt served with a 200, as plain text
curl -I https://www.example.com/robots.txt
# Simulate a training crawler: 403 expected once the Deny rule is published
curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)" https://www.example.com/
# Simulate a search crawler: 200 expected
curl -s -o /dev/null -w '%{http_code}\n' -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot)" https://www.example.com/
# X-Robots-Tag header on a document
curl -sI https://www.example.com/documents/pricing.pdf | grep -i x-robots-tag
These tests are only indicative: curl imitates the crawler's name, not its IP address, and a rule that checks where traffic comes from will not treat it as the real crawler. The check that matters happens afterwards in the project's Firewall view, where each blocked or logged request is tied to the rule that handled it. To write robots.txt rules without token mistakes, you can start from the AI robots.txt generator.
Pitfalls to avoid
- Confusing a refusal with a block. robots.txt is enough for crawlers that honor it, which OpenAI, Anthropic and Perplexity state is the case for their crawlers. The firewall is only a complement, against those that ignore it.
- Assuming a User-Agent filter stops impostors. It stops those that honestly announce their name. A scraper posing as Chrome gets through, and an overly broad filter can turn away a genuine search crawler.
- Blocking
Google-Extendedin the firewall. It is a robots.txt token, not a crawler: no request ever carries that name. BlockingGooglebotinstead would remove you from Google Search. - Forgetting subdomains and staging domains. Each one serves its own
robots.txt.
Rolling back
For robots.txt, restore the previous version of the file and redeploy; crawlers re-read the file within roughly 24 hours. For the firewall, set AI Bots Ruleset back to Allow or Bot Protection back to Off, disable or delete the custom rule, then Review Changes and Publish. For X-Robots-Tag and proxy.ts, remove the relevant block and redeploy.
Official sources
- Next.js, robots.txt (Metadata API): https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots
- Next.js, headers in next.config: https://nextjs.org/docs/app/api-reference/config/next-config-js/headers
- Next.js, proxy.js: https://nextjs.org/docs/app/api-reference/file-conventions/proxy
- Vercel, Bot Management: https://vercel.com/docs/bot-management
- Vercel, WAF Managed Rulesets: https://vercel.com/docs/vercel-firewall/vercel-waf/managed-rulesets
- Vercel, Rule Configuration Reference: https://vercel.com/docs/vercel-firewall/vercel-waf/rule-configuration
- Vercel, How to block bots from OpenAI GPTBot: https://vercel.com/kb/guide/how-to-block-bots-openai-gptbot
- Vercel, Are Vercel Preview Deployments indexed by search engines: https://vercel.com/kb/guide/are-vercel-preview-deployment-indexed-by-search-engines
- Google, Robots meta tag and X-Robots-Tag: https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag
- RFC 9309, Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309
- OpenAI, Overview of OpenAI Crawlers: https://developers.openai.com/api/docs/bots
- Anthropic, Does Anthropic crawl data from the web: https://support.claude.com/en/articles/8896518
- Perplexity, Perplexity Crawlers: https://docs.perplexity.ai/docs/resources/perplexity-crawlers
Free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
AI crawler blocklists
Ready-to-use files, refreshed hourly, to block AI training (or all AI crawlers): robots.txt, nginx, Apache, Cloudflare and the official IP ranges published by AI companies.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
Related guides
- How to Block AI Crawlers on IIS - robots.txt, URL Rewrite, Request Filtering and IP RestrictionsOn IIS, publish robots.txt, deny AI training crawlers with a URL Rewrite rule in web.config and learn the limits of request filtering and IP restrictions.
- How to Block AI Crawlers in nginx - User-Agent Rules, Official IP Ranges and Rate LimitsA tested nginx configuration that denies AI training crawlers, lets genuine AI search crawlers through, rate-limits them and writes them to a dedicated log.
- How to Block AI Crawlers in Apache 2.4 - robots.txt, .htaccess, Require not env and IP RangesDeny AI training crawlers in Apache 2.4 with .htaccess, mod_setenvif or mod_rewrite, and only accept AI search crawlers coming from their official IP ranges.
- Block or Allow AI Crawlers on WordPress - Virtual robots.txt, Physical File and PHP FilterWhere to edit robots.txt on WordPress to block GPTBot or ClaudeBot yet stay visible in ChatGPT Search, Claude and Perplexity, by file, PHP filter or plugin.