IABIGFR · Français

Platform guides · Updated 2026-11-06 · Markdown version

How to Block AI Crawlers in Apache 2.4 - robots.txt, .htaccess, Require not env and IP Ranges

Deny AI training crawlers in Apache 2.4 with .htaccess, mod_setenvif or mod_rewrite, and only accept AI search crawlers coming from their official IP ranges.

Apache HTTP Server gives you several ways to handle AI crawlers, depending on the access you have: a simple .htaccess file on shared hosting, or the full virtual host configuration on your own server. This guide covers both, with the same policy as our other guides: refuse model training, let the search crawlers of ChatGPT, Claude and Perplexity through and, where possible, deny anything that borrows their names without coming from their vendor's addresses. Every configuration below was validated with httpd -t and then with real requests on Apache 2.4.69 (the Apache Lounge Windows build); the syntax is Apache 2.4's, and a dedicated section sums up what changed from 2.2.

robots.txt is still the foundation

The file goes at the site root (the DocumentRoot). It is what expresses your choices to crawlers that respect them, and the only way to refuse training to Google (Google-Extended) or Apple (Applebot-Extended): those tokens do not correspond to any separate crawler, so no server rule can target them. A robots.txt denied with a 403 prevents the crawler from reading your refusal in it, and RFC 9309 treats a 5xx error on this file as a full disallow, so every configuration below exempts /robots.txt from blocking.

.htaccess version with mod_setenvif

This is the recommended method in 2.4: SetEnvIfNoCase sets an environment variable when the User-Agent matches, then Require not env denies requests that carry it.

# .htaccess at the site root - Apache 2.4
# Requires: AllowOverride FileInfo (SetEnvIf) and AuthConfig (Require, AuthMerging)

# 1. Denied crawlers (here: training; adjust as needed). Case-insensitive matching.
SetEnvIfNoCase User-Agent "GPTBot"             ai_deny
SetEnvIfNoCase User-Agent "ClaudeBot"          ai_deny
SetEnvIfNoCase User-Agent "CCBot"              ai_deny
SetEnvIfNoCase User-Agent "Bytespider"         ai_deny
SetEnvIfNoCase User-Agent "meta-externalagent" ai_deny

# 2. robots.txt stays readable by everyone: unset the variable
SetEnvIf Request_URI "^/robots\.txt$" !ai_deny

# 3. Combine with the access rules already defined above (logical AND)
AuthMerging And

# 4. Everyone, except flagged requests
<RequireAll>
    Require all granted
    Require not env ai_deny
</RequireAll>

Three details matter. A Require not can never authorize a request on its own: the Apache documentation requires pairing it with a positive condition, here Require all granted, inside a <RequireAll> block. Next, without AuthMerging And, the access rules in a .htaccess replace those of the parent sections: a Require all granted placed at the root could reopen a site that the server configuration restricted, for example to certain addresses. Finally, directives in a .htaccess are only read if AllowOverride permits it; since Apache 2.4, its default value is None.

.htaccess version with mod_rewrite

If the site already uses mod_rewrite, or if your host does not allow AuthConfig, the same rule can be written like this:

# .htaccess - mod_rewrite variant (AllowOverride FileInfo is enough)
RewriteEngine On
# Never block robots.txt
RewriteCond %{REQUEST_URI} !^/robots\.txt$
# [NC]: case-insensitive comparison
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent) [NC]
# [F]: 403 Forbidden response, stops rule processing
RewriteRule ^ - [F]

In our tests, both variants gave the same result: a 403 for gptbot and CCBot, a 200 for GPTBot on /robots.txt, and a 200 for OAI-SearchBot and for a browser.

Rather than maintaining the list of names by hand, you can start from the AI blocklists page, which publishes a ready-made .htaccess block (plus robots.txt, nginx and Cloudflare versions) regenerated every hour from our bot database.

Server configuration with IP range verification

On your own server, you can go further and only accept a search crawler if it comes from the ranges published by its vendor. The address lists live in separate files, generated by a script and included in the virtual host.

# Inside <VirtualHost *:443> - Apache 2.4.16 or later
# Modules: setenvif, authz_core, authz_host, log_config

# 1. Vendor claimed by the User-Agent
SetEnvIfNoCase User-Agent "GPTBot|OAI-SearchBot|ChatGPT-User"       ai_vendor=openai
SetEnvIfNoCase User-Agent "ClaudeBot|Claude-SearchBot|Claude-User" ai_vendor=anthropic
SetEnvIfNoCase User-Agent "PerplexityBot|Perplexity-User"          ai_vendor=perplexity

# 2. Crawlers denied on principle (training), except on robots.txt
SetEnvIfNoCase User-Agent "GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent" ai_deny
SetEnvIf Request_URI "^/robots\.txt$" !ai_deny

# 3. Dedicated log: only requests from an announced AI crawler
LogFormat "%h %t \"%r\" %>s vendor=%{ai_vendor}e denied=%{ai_deny}e \"%{User-Agent}i\"" ai_crawlers
CustomLog ${APACHE_LOG_DIR}/ai-crawlers.log ai_crawlers env=ai_vendor

<Location "/">
    # Does not replace the <Directory> rules: adds to them
    AuthMerging And
    <RequireAll>
        Require not env ai_deny
        <RequireAny>
            # No AI crawler announced: nothing to verify
            Require expr "-z %{ENV:ai_vendor}"
            # robots.txt always accessible
            Require expr "%{REQUEST_URI} == '/robots.txt'"
            # Announced crawler: it must come from its vendor's ranges
            <RequireAll>
                Require expr "%{ENV:ai_vendor} == 'openai'"
                Include /etc/apache2/ai-crawlers/openai.conf
            </RequireAll>
            <RequireAll>
                Require expr "%{ENV:ai_vendor} == 'anthropic'"
                Include /etc/apache2/ai-crawlers/anthropic.conf
            </RequireAll>
            <RequireAll>
                Require expr "%{ENV:ai_vendor} == 'perplexity'"
                Include /etc/apache2/ai-crawlers/perplexity.conf
            </RequireAll>
        </RequireAny>
    </RequireAll>
</Location>

Each included file contains a <RequireAny> block of Require ip lines. That block is essential: several Require ip lines placed directly inside a <RequireAll> would all have to succeed at once, which is impossible. Require ip accepts CIDR notation, for both IPv4 and IPv6. The following script, run against the official files in October 2026, produces those files and only reloads Apache if the configuration is still valid.

#!/usr/bin/env bash
# /usr/local/sbin/update-ai-crawlers-apache.sh - requires curl and jq
set -euo pipefail
dir=/etc/apache2/ai-crawlers

generate() {  # $1 = vendor, followed by the URLs of the official JSON files
  local vendor=$1; shift
  local tmp; tmp=$(mktemp)
  for url in "$@"; do
    # -L: follow redirects if a list moves
    curl -fsSL --max-time 30 "$url" \
      | jq -er '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) | select(. != null)' >> "$tmp"
  done
  [ -s "$tmp" ] || { echo "empty list for $vendor, aborting" >&2; rm -f "$tmp"; exit 1; }
  { echo "<RequireAny>"
    sort -u "$tmp" | sed 's/^/    Require ip /'
    echo "</RequireAny>"; } > "$dir/$vendor.conf.new"
  rm -f "$tmp"
}

generate openai     https://openai.com/gptbot.json https://openai.com/searchbot.json https://openai.com/chatgpt-user.json
generate anthropic  https://claude.com/crawling/bots.json
generate perplexity https://www.perplexity.ai/perplexitybot.json https://www.perplexity.ai/perplexity-user.json

for e in openai anthropic perplexity; do
  cp -p "$dir/$e.conf" "$dir/$e.conf.bak" 2>/dev/null || true
  mv "$dir/$e.conf.new" "$dir/$e.conf"
done

if ! apachectl configtest; then
  for e in openai anthropic perplexity; do
    [ -f "$dir/$e.conf.bak" ] && mv "$dir/$e.conf.bak" "$dir/$e.conf"
  done
  echo "configuration rejected, previous lists restored" >&2
  exit 1
fi
apachectl graceful

Run it once before adding the <Location> block, otherwise the Include directives point to missing files, then schedule it daily with cron. On Debian and Ubuntu, the command is also available as apache2ctl; the paths used here (/etc/apache2, ${APACHE_LOG_DIR}, which Debian defines in /etc/apache2/envvars) are Debian's, so on RHEL-family systems use /etc/httpd and an explicit log path instead. To check by hand an address found in the log, the IABIG crawler IP checker queries the same lists.

If a proxy or a CDN sits in front of Apache, the address seen by Require ip is the intermediary's: configure mod_remoteip first, otherwise every genuine crawler will be denied.

X-Robots-Tag with mod_headers

<FilesMatch "\.pdf$">
    Header set X-Robots-Tag "noindex"
</FilesMatch>

Google documents this header; the OpenAI, Anthropic and Perplexity pages we consulted for this guide do not mention it. For those crawlers, robots.txt remains the documented control.

What changed from Apache 2.2

Apache 2.2 Apache 2.4
Order Allow,Deny then Allow from all Require all granted
Order Deny,Allow then Deny from all Require all denied
Deny from env=ai_deny Require not env ai_deny inside a <RequireAll>
Allow from 192.0.2.0/24 Require ip 192.0.2.0/24
AllowOverride All by default AllowOverride None by default

The old directives still exist in 2.4 through mod_access_compat, but the documentation marks them as deprecated and advises against mixing them with Require, because the two mechanisms combine in counterintuitive ways. If you are taking over an old .htaccess, convert it entirely. One more difference: before version 2.4.16, the expression in Require expr had to be written without quotes.

Apache has no built-in equivalent of nginx's per-request rate limiting: mod_ratelimit limits bandwidth, not the number of requests.

Testing

apachectl configtest
# Training crawler: 403; same crawler on robots.txt: 200
curl -s -o /dev/null -w '%{http_code}\n' -A "compatible; GPTBot/1.4" https://www.example.com/
curl -s -o /dev/null -w '%{http_code}\n' -A "compatible; GPTBot/1.4" https://www.example.com/robots.txt
# Fake search crawler from your own machine: 403 with IP verification
curl -s -o /dev/null -w '%{http_code}\n' -A "compatible; OAI-SearchBot/1.4" https://www.example.com/
tail -f /var/log/apache2/ai-crawlers.log

These tests are only indicative: curl reproduces the User-Agent, not the crawler's address. In our tests, with the local address declared as an OpenAI range, OAI-SearchBot got a 200 and Claude-SearchBot a 403, as expected. The error log gives the reason for each denial (AH01630: client denied by server configuration).

Pitfalls to avoid

  • The User-Agent alone proves nothing. Denying by name only stops honest crawlers; only the comparison with IP ranges keeps impostors out, and only for vendors that publish them.
  • An overly broad pattern. bot would deny Googlebot and Bingbot.
  • Blocking a vendor's IP ranges to refuse training. Anthropic warns that blocking its IP addresses "may not work correctly or persistently guarantee an opt-out", because it also stops its crawlers from reading robots.txt.
  • Outdated lists. Ranges change; a search crawler denied in the log often points to a list that is too old.

Rolling back

In a .htaccess, delete or comment out the block you added; the effect is immediate. In the server configuration, remove the <Location> block and the SetEnvIfNoCase lines, then run apachectl configtest and apachectl graceful, and delete the cron job.

Official sources

Free tools

Related guides