How to Block AI Crawlers in Apache 2.4 - robots.txt, .htaccess, Require not env and IP Ranges
Deny AI training crawlers in Apache 2.4 with .htaccess, mod_setenvif or mod_rewrite, and only accept AI search crawlers coming from their official IP ranges.
Apache HTTP Server gives you several ways to handle AI crawlers, depending on the access you have: a simple .htaccess file on shared hosting, or the full virtual host configuration on your own server. This guide covers both, with the same policy as our other guides: refuse model training, let the search crawlers of ChatGPT, Claude and Perplexity through and, where possible, deny anything that borrows their names without coming from their vendor's addresses. Every configuration below was validated with httpd -t and then with real requests on Apache 2.4.69 (the Apache Lounge Windows build); the syntax is Apache 2.4's, and a dedicated section sums up what changed from 2.2.
robots.txt is still the foundation
The file goes at the site root (the DocumentRoot). It is what expresses your choices to crawlers that respect them, and the only way to refuse training to Google (Google-Extended) or Apple (Applebot-Extended): those tokens do not correspond to any separate crawler, so no server rule can target them. A robots.txt denied with a 403 prevents the crawler from reading your refusal in it, and RFC 9309 treats a 5xx error on this file as a full disallow, so every configuration below exempts /robots.txt from blocking.
.htaccess version with mod_setenvif
This is the recommended method in 2.4: SetEnvIfNoCase sets an environment variable when the User-Agent matches, then Require not env denies requests that carry it.
# .htaccess at the site root - Apache 2.4
# Requires: AllowOverride FileInfo (SetEnvIf) and AuthConfig (Require, AuthMerging)
# 1. Denied crawlers (here: training; adjust as needed). Case-insensitive matching.
SetEnvIfNoCase User-Agent "GPTBot" ai_deny
SetEnvIfNoCase User-Agent "ClaudeBot" ai_deny
SetEnvIfNoCase User-Agent "CCBot" ai_deny
SetEnvIfNoCase User-Agent "Bytespider" ai_deny
SetEnvIfNoCase User-Agent "meta-externalagent" ai_deny
# 2. robots.txt stays readable by everyone: unset the variable
SetEnvIf Request_URI "^/robots\.txt$" !ai_deny
# 3. Combine with the access rules already defined above (logical AND)
AuthMerging And
# 4. Everyone, except flagged requests
<RequireAll>
Require all granted
Require not env ai_deny
</RequireAll>
Three details matter. A Require not can never authorize a request on its own: the Apache documentation requires pairing it with a positive condition, here Require all granted, inside a <RequireAll> block. Next, without AuthMerging And, the access rules in a .htaccess replace those of the parent sections: a Require all granted placed at the root could reopen a site that the server configuration restricted, for example to certain addresses. Finally, directives in a .htaccess are only read if AllowOverride permits it; since Apache 2.4, its default value is None.
.htaccess version with mod_rewrite
If the site already uses mod_rewrite, or if your host does not allow AuthConfig, the same rule can be written like this:
# .htaccess - mod_rewrite variant (AllowOverride FileInfo is enough)
RewriteEngine On
# Never block robots.txt
RewriteCond %{REQUEST_URI} !^/robots\.txt$
# [NC]: case-insensitive comparison
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent) [NC]
# [F]: 403 Forbidden response, stops rule processing
RewriteRule ^ - [F]
In our tests, both variants gave the same result: a 403 for gptbot and CCBot, a 200 for GPTBot on /robots.txt, and a 200 for OAI-SearchBot and for a browser.
Rather than maintaining the list of names by hand, you can start from the AI blocklists page, which publishes a ready-made .htaccess block (plus robots.txt, nginx and Cloudflare versions) regenerated every hour from our bot database.
Server configuration with IP range verification
On your own server, you can go further and only accept a search crawler if it comes from the ranges published by its vendor. The address lists live in separate files, generated by a script and included in the virtual host.
# Inside <VirtualHost *:443> - Apache 2.4.16 or later
# Modules: setenvif, authz_core, authz_host, log_config
# 1. Vendor claimed by the User-Agent
SetEnvIfNoCase User-Agent "GPTBot|OAI-SearchBot|ChatGPT-User" ai_vendor=openai
SetEnvIfNoCase User-Agent "ClaudeBot|Claude-SearchBot|Claude-User" ai_vendor=anthropic
SetEnvIfNoCase User-Agent "PerplexityBot|Perplexity-User" ai_vendor=perplexity
# 2. Crawlers denied on principle (training), except on robots.txt
SetEnvIfNoCase User-Agent "GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent" ai_deny
SetEnvIf Request_URI "^/robots\.txt$" !ai_deny
# 3. Dedicated log: only requests from an announced AI crawler
LogFormat "%h %t \"%r\" %>s vendor=%{ai_vendor}e denied=%{ai_deny}e \"%{User-Agent}i\"" ai_crawlers
CustomLog ${APACHE_LOG_DIR}/ai-crawlers.log ai_crawlers env=ai_vendor
<Location "/">
# Does not replace the <Directory> rules: adds to them
AuthMerging And
<RequireAll>
Require not env ai_deny
<RequireAny>
# No AI crawler announced: nothing to verify
Require expr "-z %{ENV:ai_vendor}"
# robots.txt always accessible
Require expr "%{REQUEST_URI} == '/robots.txt'"
# Announced crawler: it must come from its vendor's ranges
<RequireAll>
Require expr "%{ENV:ai_vendor} == 'openai'"
Include /etc/apache2/ai-crawlers/openai.conf
</RequireAll>
<RequireAll>
Require expr "%{ENV:ai_vendor} == 'anthropic'"
Include /etc/apache2/ai-crawlers/anthropic.conf
</RequireAll>
<RequireAll>
Require expr "%{ENV:ai_vendor} == 'perplexity'"
Include /etc/apache2/ai-crawlers/perplexity.conf
</RequireAll>
</RequireAny>
</RequireAll>
</Location>
Each included file contains a <RequireAny> block of Require ip lines. That block is essential: several Require ip lines placed directly inside a <RequireAll> would all have to succeed at once, which is impossible. Require ip accepts CIDR notation, for both IPv4 and IPv6. The following script, run against the official files in October 2026, produces those files and only reloads Apache if the configuration is still valid.
#!/usr/bin/env bash
# /usr/local/sbin/update-ai-crawlers-apache.sh - requires curl and jq
set -euo pipefail
dir=/etc/apache2/ai-crawlers
generate() { # $1 = vendor, followed by the URLs of the official JSON files
local vendor=$1; shift
local tmp; tmp=$(mktemp)
for url in "$@"; do
# -L: follow redirects if a list moves
curl -fsSL --max-time 30 "$url" \
| jq -er '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) | select(. != null)' >> "$tmp"
done
[ -s "$tmp" ] || { echo "empty list for $vendor, aborting" >&2; rm -f "$tmp"; exit 1; }
{ echo "<RequireAny>"
sort -u "$tmp" | sed 's/^/ Require ip /'
echo "</RequireAny>"; } > "$dir/$vendor.conf.new"
rm -f "$tmp"
}
generate openai https://openai.com/gptbot.json https://openai.com/searchbot.json https://openai.com/chatgpt-user.json
generate anthropic https://claude.com/crawling/bots.json
generate perplexity https://www.perplexity.ai/perplexitybot.json https://www.perplexity.ai/perplexity-user.json
for e in openai anthropic perplexity; do
cp -p "$dir/$e.conf" "$dir/$e.conf.bak" 2>/dev/null || true
mv "$dir/$e.conf.new" "$dir/$e.conf"
done
if ! apachectl configtest; then
for e in openai anthropic perplexity; do
[ -f "$dir/$e.conf.bak" ] && mv "$dir/$e.conf.bak" "$dir/$e.conf"
done
echo "configuration rejected, previous lists restored" >&2
exit 1
fi
apachectl graceful
Run it once before adding the <Location> block, otherwise the Include directives point to missing files, then schedule it daily with cron. On Debian and Ubuntu, the command is also available as apache2ctl; the paths used here (/etc/apache2, ${APACHE_LOG_DIR}, which Debian defines in /etc/apache2/envvars) are Debian's, so on RHEL-family systems use /etc/httpd and an explicit log path instead. To check by hand an address found in the log, the IABIG crawler IP checker queries the same lists.
If a proxy or a CDN sits in front of Apache, the address seen by Require ip is the intermediary's: configure mod_remoteip first, otherwise every genuine crawler will be denied.
X-Robots-Tag with mod_headers
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex"
</FilesMatch>
Google documents this header; the OpenAI, Anthropic and Perplexity pages we consulted for this guide do not mention it. For those crawlers, robots.txt remains the documented control.
What changed from Apache 2.2
| Apache 2.2 | Apache 2.4 |
|---|---|
Order Allow,Deny then Allow from all |
Require all granted |
Order Deny,Allow then Deny from all |
Require all denied |
Deny from env=ai_deny |
Require not env ai_deny inside a <RequireAll> |
Allow from 192.0.2.0/24 |
Require ip 192.0.2.0/24 |
AllowOverride All by default |
AllowOverride None by default |
The old directives still exist in 2.4 through mod_access_compat, but the documentation marks them as deprecated and advises against mixing them with Require, because the two mechanisms combine in counterintuitive ways. If you are taking over an old .htaccess, convert it entirely. One more difference: before version 2.4.16, the expression in Require expr had to be written without quotes.
Apache has no built-in equivalent of nginx's per-request rate limiting: mod_ratelimit limits bandwidth, not the number of requests.
Testing
apachectl configtest
# Training crawler: 403; same crawler on robots.txt: 200
curl -s -o /dev/null -w '%{http_code}\n' -A "compatible; GPTBot/1.4" https://www.example.com/
curl -s -o /dev/null -w '%{http_code}\n' -A "compatible; GPTBot/1.4" https://www.example.com/robots.txt
# Fake search crawler from your own machine: 403 with IP verification
curl -s -o /dev/null -w '%{http_code}\n' -A "compatible; OAI-SearchBot/1.4" https://www.example.com/
tail -f /var/log/apache2/ai-crawlers.log
These tests are only indicative: curl reproduces the User-Agent, not the crawler's address. In our tests, with the local address declared as an OpenAI range, OAI-SearchBot got a 200 and Claude-SearchBot a 403, as expected. The error log gives the reason for each denial (AH01630: client denied by server configuration).
Pitfalls to avoid
- The User-Agent alone proves nothing. Denying by name only stops honest crawlers; only the comparison with IP ranges keeps impostors out, and only for vendors that publish them.
- An overly broad pattern.
botwould deny Googlebot and Bingbot. - Blocking a vendor's IP ranges to refuse training. Anthropic warns that blocking its IP addresses "may not work correctly or persistently guarantee an opt-out", because it also stops its crawlers from reading robots.txt.
- Outdated lists. Ranges change; a search crawler denied in the log often points to a list that is too old.
Rolling back
In a .htaccess, delete or comment out the block you added; the effect is immediate. In the server configuration, remove the <Location> block and the SetEnvIfNoCase lines, then run apachectl configtest and apachectl graceful, and delete the cron job.
Official sources
- Apache 2.4, mod_authz_core (Require, RequireAll, AuthMerging): https://httpd.apache.org/docs/2.4/mod/mod_authz_core.html
- Apache 2.4, mod_authz_host (Require ip): https://httpd.apache.org/docs/2.4/mod/mod_authz_host.html
- Apache 2.4, mod_setenvif: https://httpd.apache.org/docs/2.4/mod/mod_setenvif.html
- Apache 2.4, mod_rewrite: https://httpd.apache.org/docs/2.4/mod/mod_rewrite.html
- Apache 2.4, Access Control: https://httpd.apache.org/docs/2.4/howto/access.html
- Apache 2.4, Upgrading to 2.4 from 2.2: https://httpd.apache.org/docs/2.4/upgrading.html
- Apache 2.4, mod_log_config (CustomLog env=): https://httpd.apache.org/docs/2.4/mod/mod_log_config.html
- Apache 2.4, mod_headers: https://httpd.apache.org/docs/2.4/mod/mod_headers.html
- Apache 2.4, mod_remoteip: https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html
- Apache 2.4, mod_ratelimit: https://httpd.apache.org/docs/2.4/mod/mod_ratelimit.html
- RFC 9309, Robots Exclusion Protocol: https://www.rfc-editor.org/rfc/rfc9309
- OpenAI, Overview of OpenAI Crawlers: https://developers.openai.com/api/docs/bots
- Anthropic, Does Anthropic crawl data from the web: https://support.claude.com/en/articles/8896518
- Perplexity, Perplexity Crawlers: https://docs.perplexity.ai/docs/resources/perplexity-crawlers
Free tools
Free AI visibility checker
Check for free whether ChatGPT, Claude, Perplexity and Gemini can read and cite your website - robots.txt, llms.txt, JavaScript and hidden text.
AI robots.txt generator
Build a robots.txt in one click that blocks AI training while keeping your site visible in ChatGPT Search, Claude and Perplexity. Free.
AI crawler blocklists
Ready-to-use files, refreshed hourly, to block AI training (or all AI crawlers): robots.txt, nginx, Apache, Cloudflare and the official IP ranges published by AI companies.
robots.txt tester for AI crawlers
Paste your robots.txt and instantly see, crawler by crawler, whether GPTBot, ClaudeBot, PerplexityBot or Google-Extended can access a page.
Related guides
- How to Block AI Crawlers on IIS - robots.txt, URL Rewrite, Request Filtering and IP RestrictionsOn IIS, publish robots.txt, deny AI training crawlers with a URL Rewrite rule in web.config and learn the limits of request filtering and IP restrictions.
- How to Block AI Crawlers in nginx - User-Agent Rules, Official IP Ranges and Rate LimitsA tested nginx configuration that denies AI training crawlers, lets genuine AI search crawlers through, rate-limits them and writes them to a dedicated log.
- Block or Allow AI Crawlers on WordPress - Virtual robots.txt, Physical File and PHP FilterWhere to edit robots.txt on WordPress to block GPTBot or ClaudeBot yet stay visible in ChatGPT Search, Claude and Perplexity, by file, PHP filter or plugin.
- Block or Allow AI Crawlers on Wix - The robots.txt Editor and Its LimitsEdit robots.txt on Wix to block GPTBot or ClaudeBot without blocking ChatGPT Search, Claude and Perplexity, where to find the editor and the pitfalls to avoid.