# Block or Allow AI Crawlers on WordPress - Virtual robots.txt, Physical File and PHP Filter

> Where to edit robots.txt on WordPress to block GPTBot or ClaudeBot yet stay visible in ChatGPT Search, Claude and Perplexity, by file, PHP filter or plugin.

Source: https://iabig.com/guides/platforms/wordpress-ai-crawlers/ — Guides CC BY 4.0, extraits de code MIT

On a self-hosted WordPress site, the `robots.txt` file can come from three places: WordPress itself, which generates it on the fly; a real file uploaded to the site root; or an SEO plugin that rewrites the generated version. Finding out which one is actually being served is the first step, because a change made in the wrong place has no effect. This guide walks through each method, gives you a ready-to-copy file that blocks model training while letting the search crawlers of ChatGPT, Claude and Perplexity through, and lists the pitfalls specific to WordPress. It does not cover sites hosted on WordPress.com, which have their own settings.

## How WordPress generates its robots.txt

As long as no `robots.txt` file exists at the root, WordPress answers requests to `/robots.txt` itself, through the `do_robots()` function. The default output is minimal:

```text
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/wp-sitemap.xml
```

The `Sitemap` line has been added since WordPress 5.5, and only when the site is public. An SEO plugin that manages its own sitemap may replace it.

Before it is sent, this text goes through the `robots_txt` filter (available since WordPress 3.0), which receives two parameters: the generated content and a boolean that says whether the site is public. SEO plugins use this filter, and so can you.

As soon as a real `robots.txt` file exists at the root, the web server serves it directly. WordPress is never called, the filter does not run, and the robots.txt editors in SEO plugins stop having any effect. The WordPress documentation also points out that the virtual version only works when WordPress is installed at the root of the domain.

## The "Search Engine Visibility" setting

Under **Settings > Reading**, the "Discourage search engines from indexing this site" checkbox is not a way to sort AI crawlers. Since WordPress 5.3, it adds a `<meta name="robots" content="noindex,nofollow">` tag to the page head and no longer changes `robots.txt`; up to version 5.2, it served `Disallow: /` to every crawler. Either way, it is a blanket request sent to all search engines. If you want to appear in AI assistant answers, leave this box unchecked and handle AI crawlers in `robots.txt`.

## Pick a single method

Three methods coexist, and mixing them is the most common source of mistakes:

- **A physical file** uploaded by FTP or through your host's file manager. Simple and easy to read, but it overrides everything else.
- **A PHP filter** in a must-use plugin or in `functions.php`. It keeps the version generated by WordPress, including the `Sitemap` line.
- **The editor in an SEO plugin** such as Yoast SEO or Rank Math, convenient if the plugin is already installed.

Choose one and stick with it. A custom filter and an SEO plugin both modify the same text, so the result then depends on the order in which they run.

## A ready-to-copy robots.txt

This file blocks training, allows search, and keeps the WordPress default rules. Remove from the first group any crawler you are willing to accept.

```text
# 1. Model training: blocked
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: Applebot-Extended
User-agent: Google-Extended
Disallow: /

# 2. Search and on-demand retrieval: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

# 3. All other crawlers
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.example.com/wp-sitemap.xml
```

Group 2 repeats the rules of the `*` group because a crawler that finds a group with its own name follows only that group. Adjust the sitemap URL: copy the one currently listed in your `robots.txt`, since it depends on which plugin generates the sitemap. Finally, `Google-Extended` also covers grounding of answers in the Gemini apps, so remove that line if you want to stay visible there.

## The PHP filter method

To keep the version generated by WordPress and add your own groups to it, create the file `wp-content/mu-plugins/ai-crawlers.php`. Plugins in this folder are activated automatically; WordPress only loads PHP files placed directly in `mu-plugins`, not in a subfolder.

```php
<?php
/**
 * Plugin Name: AI Crawlers
 * Description: Adds groups for AI crawlers to the robots.txt generated by WordPress.
 */

add_filter( 'robots_txt', function ( $output, $public ) {
	// Site hidden from search engines: change nothing.
	if ( ! $public ) {
		return $output;
	}

	$admin = wp_parse_url( admin_url(), PHP_URL_PATH );
	$ajax  = wp_parse_url( admin_url( 'admin-ajax.php' ), PHP_URL_PATH );

	$training = array( 'GPTBot', 'ClaudeBot', 'CCBot', 'meta-externalagent',
		'MistralAI-Training', 'Applebot-Extended', 'Google-Extended' );
	$search   = array( 'OAI-SearchBot', 'ChatGPT-User', 'Claude-SearchBot',
		'Claude-User', 'PerplexityBot', 'Perplexity-User' );

	$ai = '';
	foreach ( $training as $robot ) {
		$ai .= "User-agent: {$robot}\n";
	}
	$ai .= "Disallow: /\n\n";

	foreach ( $search as $robot ) {
		$ai .= "User-agent: {$robot}\n";
	}
	$ai .= "Disallow: {$admin}\nAllow: {$ajax}\n\n";

	return $ai . $output;
}, 20, 2 );
```

The code puts your groups before the existing content without touching the `*` group or the `Sitemap` line, and it computes the admin path instead of hard-coding it. You can also paste the function into the `functions.php` file of a child theme. Avoid the parent theme, whose file can be overwritten by an update, and keep in mind that the rule disappears if you switch themes.

## With Yoast SEO or Rank Math

**Yoast SEO.** The editor is under **Yoast SEO > Tools > File editor**, followed by "Create robots.txt file" if no file exists yet. Yoast notes that this menu does not appear when file editing is disabled in WordPress (which is what the `DISALLOW_FILE_EDIT` constant in `wp-config.php` does); in that case, edit the file over FTP instead. The plugin also has a blocking feature under **Yoast SEO > Settings > Advanced > Crawl optimization**, in the "Block unwanted bots" section. According to Yoast, the free version can only block Google AdsBot; Premium adds Google-Extended (Gemini and Vertex AI), GPTBot and CCBot. These toggles only block: they do not create an allow group for search crawlers, so ClaudeBot, Perplexity's crawlers and the rest still need manual rules.

**Rank Math.** The editor is under **Rank Math SEO > General Settings > Edit robots.txt**. It only appears in Advanced Mode, which you can switch on from the plugin's dashboard. Rank Math generates a virtual file: if a physical file exists, Rank Math shows a notice and you cannot edit it from the plugin until you delete that file (by FTP or your host's file manager).

## WordPress-specific pitfalls

- **A forgotten physical file.** An uninstalled plugin or your host may have left a `robots.txt` at the root. As long as it exists, your filter and the SEO plugin's editor do nothing.
- **Caching.** A caching plugin or a CDN may keep serving the old version. Purge them after every change.
- **An install in a subdirectory.** If WordPress lives in `/blog/`, its virtual `robots.txt` is not at the root of the domain, which is the only place crawlers look. You then need a file at the root.
- **Forgetting the admin rules.** Putting only `Allow: /` in the search crawler group opens `/wp-admin/` to them. Not really dangerous (the area requires a login), but pointless.
- **The firewall.** A security plugin or web application firewall that blocks AI crawlers makes your `robots.txt` irrelevant for them. Also check that `/robots.txt` does not return a 5xx error, which the standard treats as a full disallow.

## Check the result

Open `https://www.example.com/robots.txt` in a private browsing window: you should see your groups, exactly once. From the command line, `curl -I https://www.example.com/robots.txt` should return a 200 status code and a `text/plain` content type. Then check, crawler by crawler, which URLs are allowed with the [IABIG robots.txt tester](/tools/robots-txt-tester/). Crawlers cache robots.txt rather than re-reading it on every request; RFC 9309 asks them not to keep a cached copy for more than 24 hours, so give changes a day before judging the result.

## Official sources

- WordPress, `do_robots()`: <https://developer.wordpress.org/reference/functions/do_robots/>
- WordPress, `robots_txt` filter: <https://developer.wordpress.org/reference/hooks/robots_txt/>
- WordPress, `WP_Sitemaps::add_robots()`: <https://developer.wordpress.org/reference/classes/wp_sitemaps/add_robots/>
- WordPress, Settings Reading Screen: <https://wordpress.org/documentation/article/settings-reading-screen/>
- WordPress, Must Use Plugins: <https://developer.wordpress.org/advanced-administration/plugins/mu-plugins/>
- Yoast, How to edit robots.txt through Yoast SEO: <https://yoast.com/help/how-to-edit-robots-txt-through-yoast-seo/>
- Yoast, How to block unwanted bots from your site with Yoast SEO: <https://yoast.com/help/block-unwanted-bots-with-yoast-seo/>
- Rank Math, Common robots.txt issues: <https://rankmath.com/kb/fix-common-robots-txt-issues/>
- Rank Math, I can't edit my website's robots.txt: <https://rankmath.com/kb/cant-edit-robots-txt/>
- RFC 9309, Robots Exclusion Protocol: <https://www.rfc-editor.org/rfc/rfc9309>
