Block or Allow AI Crawlers on WordPress - Virtual robots.txt, Physical File and PHP Filter
Where to edit robots.txt on WordPress to block GPTBot or ClaudeBot yet stay visible in ChatGPT Search, Claude and Perplexity, by file, PHP filter or plugin.
On a self-hosted WordPress site, the robots.txt file can come from three places: WordPress itself, which generates it on the fly; a real file uploaded to the site root; or an SEO plugin that rewrites the generated version. Finding out which one is actually being served is the first step, because a change made in the wrong place has no effect. This guide walks through each method, gives you a ready-to-copy file that blocks model training while letting the search crawlers of ChatGPT, Claude and Perplexity through, and lists the pitfalls specific to WordPress. It does not cover sites hosted on WordPress.com, which have their own settings.
How WordPress generates its robots.txt
As long as no robots.txt file exists at the root, WordPress answers requests to /robots.txt itself, through the do_robots() function. The default output is minimal:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/wp-sitemap.xml
The Sitemap line has been added since WordPress 5.5, and only when the site is public. An SEO plugin that manages its own sitemap may replace it.
Before it is sent, this text goes through the robots_txt filter (available since WordPress 3.0), which receives two parameters: the generated content and a boolean that says whether the site is public. SEO plugins use this filter, and so can you.
As soon as a real robots.txt file exists at the root, the web server serves it directly. WordPress is never called, the filter does not run, and the robots.txt editors in SEO plugins stop having any effect. The WordPress documentation also points out that the virtual version only works when WordPress is installed at the root of the domain.
The "Search Engine Visibility" setting
Under Settings > Reading, the "Discourage search engines from indexing this site" checkbox is not a way to sort AI crawlers. Since WordPress 5.3, it adds a <meta name="robots" content="noindex,nofollow"> tag to the page head and no longer changes robots.txt; up to version 5.2, it served Disallow: / to every crawler. Either way, it is a blanket request sent to all search engines. If you want to appear in AI assistant answers, leave this box unchecked and handle AI crawlers in robots.txt.
Pick a single method
Three methods coexist, and mixing them is the most common source of mistakes:
- A physical file uploaded by FTP or through your host's file manager. Simple and easy to read, but it overrides everything else.
- A PHP filter in a must-use plugin or in
functions.php. It keeps the version generated by WordPress, including theSitemapline. - The editor in an SEO plugin such as Yoast SEO or Rank Math, convenient if the plugin is already installed.
Choose one and stick with it. A custom filter and an SEO plugin both modify the same text, so the result then depends on the order in which they run.
A ready-to-copy robots.txt
This file blocks training, allows search, and keeps the WordPress default rules. Remove from the first group any crawler you are willing to accept.
# 1. Model training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Applebot-Extended
User-agent: Google-Extended
Disallow: /
# 2. Search and on-demand retrieval: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# 3. All other crawlers
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.example.com/wp-sitemap.xml
Group 2 repeats the rules of the * group because a crawler that finds a group with its own name follows only that group. Adjust the sitemap URL: copy the one currently listed in your robots.txt, since it depends on which plugin generates the sitemap. Finally, Google-Extended also covers grounding of answers in the Gemini apps, so remove that line if you want to stay visible there.
The PHP filter method
To keep the version generated by WordPress and add your own groups to it, create the file wp-content/mu-plugins/ai-crawlers.php. Plugins in this folder are activated automatically; WordPress only loads PHP files placed directly in mu-plugins, not in a subfolder.
<?php
/**
* Plugin Name: AI Crawlers
* Description: Adds groups for AI crawlers to the robots.txt generated by WordPress.
*/
add_filter( 'robots_txt', function ( $output, $public ) {
// Site hidden from search engines: change nothing.
if ( ! $public ) {
return $output;
}
$admin = wp_parse_url( admin_url(), PHP_URL_PATH );
$ajax = wp_parse_url( admin_url( 'admin-ajax.php' ), PHP_URL_PATH );
$training = array( 'GPTBot', 'ClaudeBot', 'CCBot', 'meta-externalagent',
'MistralAI-Training', 'Applebot-Extended', 'Google-Extended' );
$search = array( 'OAI-SearchBot', 'ChatGPT-User', 'Claude-SearchBot',
'Claude-User', 'PerplexityBot', 'Perplexity-User' );
$ai = '';
foreach ( $training as $robot ) {
$ai .= "User-agent: {$robot}\n";
}
$ai .= "Disallow: /\n\n";
foreach ( $search as $robot ) {
$ai .= "User-agent: {$robot}\n";
}
$ai .= "Disallow: {$admin}\nAllow: {$ajax}\n\n";
return $ai . $output;
}, 20, 2 );
The code puts your groups before the existing content without touching the * group or the Sitemap line, and it computes the admin path instead of hard-coding it. You can also paste the function into the functions.php file of a child theme. Avoid the parent theme, whose file can be overwritten by an update, and keep in mind that the rule disappears if you switch themes.
With Yoast SEO or Rank Math
Yoast SEO. The editor is under Yoast SEO > Tools > File editor, followed by "Create robots.txt file" if no file exists yet. Yoast notes that this menu does not appear when file editing is disabled in WordPress (which is what the DISALLOW_FILE_EDIT constant in wp-config.php does); in that case, edit the file over FTP instead. The plugin also has a blocking feature under Yoast SEO > Settings > Advanced > Crawl optimization, in the "Block unwanted bots" section. According to Yoast, the free version can only block Google AdsBot; Premium adds Google-Extended (Gemini and Vertex AI), GPTBot and CCBot. These toggles only block: they do not create an allow group for search crawlers, so ClaudeBot, Perplexity's crawlers and the rest still need manual rules.
Rank Math. The editor is under Rank Math SEO > General Settings > Edit robots.txt. It only appears in Advanced Mode, which you can switch on from the plugin's dashboard. Rank Math generates a virtual file: if a physical file exists, Rank Math shows a notice and you cannot edit it from the plugin until you delete that file (by FTP or your host's file manager).
WordPress-specific pitfalls
- A forgotten physical file. An uninstalled plugin or your host may have left a
robots.txtat the root. As long as it exists, your filter and the SEO plugin's editor do nothing. - Caching. A caching plugin or a CDN may keep serving the old version. Purge them after every change.
- An install in a subdirectory. If WordPress lives in
/blog/, its virtualrobots.txtis not at the root of the domain, which is the only place crawlers look. You then need a file at the root. - Forgetting the admin rules. Putting only
Allow: /in the search crawler group opens/wp-admin/to them. Not really dangerous (the area requires a login), but pointless. - The firewall. A security plugin or web application firewall that blocks AI crawlers makes your
robots.txtirrelevant for them. Also check that/robots.txtdoes not return a 5xx error, which the standard treats as a full disallow.
Check the result
Open https://www.example.com/robots.txt in a private browsing window: you should see your groups, exactly once. From the command line, curl -I https://www.example.com/robots.txt should return a 200 status code and a text/plain content type. Then check, crawler by crawler, which URLs are allowed with the IABIG robots.txt tester. Crawlers cache robots.txt rather than re-reading it on every request; RFC 9309 asks them not to keep a cached copy for more than 24 hours, so give changes a day before judging the result.
Official sources
- WordPress,
do_robots(): https://developer.wordpress.org/reference/functions/do_robots/ - WordPress,
robots_txtfilter: https://developer.wordpress.org/reference/hooks/robots_txt/ - WordPress,
WP_Sitemaps::add_robots(): https://developer.wordpress.org/reference/classes/wp_sitemaps/add_robots/ - WordPress, Settings Reading Screen: https://wordpress.org/documentation/article/settings-reading-screen/
- WordPress, Must Use Plugins: https://developer.wordpress.org/advanced-administration/plugins/mu-plugins/
- Yoast, How to edit robots.txt through Yoast SEO: https://yoast.com/help/how-to-edit-robots-txt-through-yoast-seo/
- Yoast, How to block unwanted bots from your site with Yoast SEO: https://yoast.com/help/block-unwanted-bots-with-yoast-seo/
- Rank Math, Common robots.txt issues: https://rankmath.com/kb/fix-common-robots-txt-issues/
- Rank Math, I can't edit my website's robots.txt: https://rankmath.com/kb/cant-edit-robots-txt/
- RFC 9309, Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309
Free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
AI crawler blocklists
Ready-to-use files, refreshed hourly, to block AI training (or all AI crawlers): robots.txt, nginx, Apache, Cloudflare and the official IP ranges published by AI companies.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
Related guides
- How to Block AI Crawlers on IIS - robots.txt, URL Rewrite, Request Filtering and IP RestrictionsOn IIS, publish robots.txt, deny AI training crawlers with a URL Rewrite rule in web.config and learn the limits of request filtering and IP restrictions.
- How to Block AI Crawlers in nginx - User-Agent Rules, Official IP Ranges and Rate LimitsA tested nginx configuration that denies AI training crawlers, lets genuine AI search crawlers through, rate-limits them and writes them to a dedicated log.
- How to Block AI Crawlers in Apache 2.4 - robots.txt, .htaccess, Require not env and IP RangesDeny AI training crawlers in Apache 2.4 with .htaccess, mod_setenvif or mod_rewrite, and only accept AI search crawlers coming from their official IP ranges.
- Block or Allow AI Crawlers on Wix - The robots.txt Editor and Its LimitsEdit robots.txt on Wix to block GPTBot or ClaudeBot without blocking ChatGPT Search, Claude and Perplexity, where to find the editor and the pitfalls to avoid.