# How to Block AI Crawlers in Apache 2.4 - robots.txt, .htaccess, Require not env and IP Ranges

> Deny AI training crawlers in Apache 2.4 with .htaccess, mod_setenvif or mod_rewrite, and only accept AI search crawlers coming from their official IP ranges.

Source: https://iabig.com/guides/platforms/apache-ai-crawlers/ — Guides CC BY 4.0, extraits de code MIT

Apache HTTP Server gives you several ways to handle AI crawlers, depending on the access you have: a simple `.htaccess` file on shared hosting, or the full virtual host configuration on your own server. This guide covers both, with the same policy as our other guides: refuse model training, let the search crawlers of ChatGPT, Claude and Perplexity through and, where possible, deny anything that borrows their names without coming from their vendor's addresses. Every configuration below was validated with `httpd -t` and then with real requests on Apache 2.4.69 (the Apache Lounge Windows build); the syntax is Apache 2.4's, and a dedicated section sums up what changed from 2.2.

## robots.txt is still the foundation

The file goes at the site root (the `DocumentRoot`). It is what expresses your choices to crawlers that respect them, and the only way to refuse training to Google (`Google-Extended`) or Apple (`Applebot-Extended`): those tokens do not correspond to any separate crawler, so no server rule can target them. A robots.txt denied with a 403 prevents the crawler from reading your refusal in it, and RFC 9309 treats a 5xx error on this file as a full disallow, so every configuration below exempts `/robots.txt` from blocking.

## .htaccess version with mod_setenvif

This is the recommended method in 2.4: `SetEnvIfNoCase` sets an environment variable when the User-Agent matches, then `Require not env` denies requests that carry it.

```apache
# .htaccess at the site root - Apache 2.4
# Requires: AllowOverride FileInfo (SetEnvIf) and AuthConfig (Require, AuthMerging)

# 1. Denied crawlers (here: training; adjust as needed). Case-insensitive matching.
SetEnvIfNoCase User-Agent "GPTBot"             ai_deny
SetEnvIfNoCase User-Agent "ClaudeBot"          ai_deny
SetEnvIfNoCase User-Agent "CCBot"              ai_deny
SetEnvIfNoCase User-Agent "Bytespider"         ai_deny
SetEnvIfNoCase User-Agent "meta-externalagent" ai_deny

# 2. robots.txt stays readable by everyone: unset the variable
SetEnvIf Request_URI "^/robots\.txt$" !ai_deny

# 3. Combine with the access rules already defined above (logical AND)
AuthMerging And

# 4. Everyone, except flagged requests
<RequireAll>
    Require all granted
    Require not env ai_deny
</RequireAll>
```

Three details matter. A `Require not` can never authorize a request on its own: the Apache documentation requires pairing it with a positive condition, here `Require all granted`, inside a `<RequireAll>` block. Next, without `AuthMerging And`, the access rules in a `.htaccess` replace those of the parent sections: a `Require all granted` placed at the root could reopen a site that the server configuration restricted, for example to certain addresses. Finally, directives in a `.htaccess` are only read if `AllowOverride` permits it; since Apache 2.4, its default value is `None`.

## .htaccess version with mod_rewrite

If the site already uses `mod_rewrite`, or if your host does not allow `AuthConfig`, the same rule can be written like this:

```apache
# .htaccess - mod_rewrite variant (AllowOverride FileInfo is enough)
RewriteEngine On
# Never block robots.txt
RewriteCond %{REQUEST_URI} !^/robots\.txt$
# [NC]: case-insensitive comparison
RewriteCond %{HTTP_USER_AGENT} (GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent) [NC]
# [F]: 403 Forbidden response, stops rule processing
RewriteRule ^ - [F]
```

In our tests, both variants gave the same result: a 403 for `gptbot` and `CCBot`, a 200 for `GPTBot` on `/robots.txt`, and a 200 for `OAI-SearchBot` and for a browser.

Rather than maintaining the list of names by hand, you can start from the [AI blocklists page](/tools/ai-blocklists/), which publishes a ready-made `.htaccess` block (plus robots.txt, nginx and Cloudflare versions) regenerated every hour from our bot database.

## Server configuration with IP range verification

On your own server, you can go further and only accept a search crawler if it comes from the ranges published by its vendor. The address lists live in separate files, generated by a script and included in the virtual host.

```apache
# Inside <VirtualHost *:443> - Apache 2.4.16 or later
# Modules: setenvif, authz_core, authz_host, log_config

# 1. Vendor claimed by the User-Agent
SetEnvIfNoCase User-Agent "GPTBot|OAI-SearchBot|ChatGPT-User"       ai_vendor=openai
SetEnvIfNoCase User-Agent "ClaudeBot|Claude-SearchBot|Claude-User" ai_vendor=anthropic
SetEnvIfNoCase User-Agent "PerplexityBot|Perplexity-User"          ai_vendor=perplexity

# 2. Crawlers denied on principle (training), except on robots.txt
SetEnvIfNoCase User-Agent "GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent" ai_deny
SetEnvIf Request_URI "^/robots\.txt$" !ai_deny

# 3. Dedicated log: only requests from an announced AI crawler
LogFormat "%h %t \"%r\" %>s vendor=%{ai_vendor}e denied=%{ai_deny}e \"%{User-Agent}i\"" ai_crawlers
CustomLog ${APACHE_LOG_DIR}/ai-crawlers.log ai_crawlers env=ai_vendor

<Location "/">
    # Does not replace the <Directory> rules: adds to them
    AuthMerging And
    <RequireAll>
        Require not env ai_deny
        <RequireAny>
            # No AI crawler announced: nothing to verify
            Require expr "-z %{ENV:ai_vendor}"
            # robots.txt always accessible
            Require expr "%{REQUEST_URI} == '/robots.txt'"
            # Announced crawler: it must come from its vendor's ranges
            <RequireAll>
                Require expr "%{ENV:ai_vendor} == 'openai'"
                Include /etc/apache2/ai-crawlers/openai.conf
            </RequireAll>
            <RequireAll>
                Require expr "%{ENV:ai_vendor} == 'anthropic'"
                Include /etc/apache2/ai-crawlers/anthropic.conf
            </RequireAll>
            <RequireAll>
                Require expr "%{ENV:ai_vendor} == 'perplexity'"
                Include /etc/apache2/ai-crawlers/perplexity.conf
            </RequireAll>
        </RequireAny>
    </RequireAll>
</Location>
```

Each included file contains a `<RequireAny>` block of `Require ip` lines. That block is essential: several `Require ip` lines placed directly inside a `<RequireAll>` would all have to succeed at once, which is impossible. `Require ip` accepts CIDR notation, for both IPv4 and IPv6. The following script, run against the official files in October 2026, produces those files and only reloads Apache if the configuration is still valid.

```bash
#!/usr/bin/env bash
# /usr/local/sbin/update-ai-crawlers-apache.sh - requires curl and jq
set -euo pipefail
dir=/etc/apache2/ai-crawlers

generate() {  # $1 = vendor, followed by the URLs of the official JSON files
  local vendor=$1; shift
  local tmp; tmp=$(mktemp)
  for url in "$@"; do
    # -L: follow redirects if a list moves
    curl -fsSL --max-time 30 "$url" \
      | jq -er '.prefixes[] | (.ipv4Prefix // .ipv6Prefix) | select(. != null)' >> "$tmp"
  done
  [ -s "$tmp" ] || { echo "empty list for $vendor, aborting" >&2; rm -f "$tmp"; exit 1; }
  { echo "<RequireAny>"
    sort -u "$tmp" | sed 's/^/    Require ip /'
    echo "</RequireAny>"; } > "$dir/$vendor.conf.new"
  rm -f "$tmp"
}

generate openai     https://openai.com/gptbot.json https://openai.com/searchbot.json https://openai.com/chatgpt-user.json
generate anthropic  https://claude.com/crawling/bots.json
generate perplexity https://www.perplexity.ai/perplexitybot.json https://www.perplexity.ai/perplexity-user.json

for e in openai anthropic perplexity; do
  cp -p "$dir/$e.conf" "$dir/$e.conf.bak" 2>/dev/null || true
  mv "$dir/$e.conf.new" "$dir/$e.conf"
done

if ! apachectl configtest; then
  for e in openai anthropic perplexity; do
    [ -f "$dir/$e.conf.bak" ] && mv "$dir/$e.conf.bak" "$dir/$e.conf"
  done
  echo "configuration rejected, previous lists restored" >&2
  exit 1
fi
apachectl graceful
```

Run it once before adding the `<Location>` block, otherwise the `Include` directives point to missing files, then schedule it daily with cron. On Debian and Ubuntu, the command is also available as `apache2ctl`; the paths used here (`/etc/apache2`, `${APACHE_LOG_DIR}`, which Debian defines in `/etc/apache2/envvars`) are Debian's, so on RHEL-family systems use `/etc/httpd` and an explicit log path instead. To check by hand an address found in the log, the [IABIG crawler IP checker](/tools/verify-bot-ip/) queries the same lists.

If a proxy or a CDN sits in front of Apache, the address seen by `Require ip` is the intermediary's: configure `mod_remoteip` first, otherwise every genuine crawler will be denied.

## X-Robots-Tag with mod_headers

```apache
<FilesMatch "\.pdf$">
    Header set X-Robots-Tag "noindex"
</FilesMatch>
```

Google documents this header; the OpenAI, Anthropic and Perplexity pages we consulted for this guide do not mention it. For those crawlers, robots.txt remains the documented control.

## What changed from Apache 2.2

| Apache 2.2 | Apache 2.4 |
|---|---|
| `Order Allow,Deny` then `Allow from all` | `Require all granted` |
| `Order Deny,Allow` then `Deny from all` | `Require all denied` |
| `Deny from env=ai_deny` | `Require not env ai_deny` inside a `<RequireAll>` |
| `Allow from 192.0.2.0/24` | `Require ip 192.0.2.0/24` |
| `AllowOverride All` by default | `AllowOverride None` by default |

The old directives still exist in 2.4 through `mod_access_compat`, but the documentation marks them as deprecated and advises against mixing them with `Require`, because the two mechanisms combine in counterintuitive ways. If you are taking over an old `.htaccess`, convert it entirely. One more difference: before version 2.4.16, the expression in `Require expr` had to be written without quotes.

Apache has no built-in equivalent of nginx's per-request rate limiting: `mod_ratelimit` limits bandwidth, not the number of requests.

## Testing

```bash
apachectl configtest
# Training crawler: 403; same crawler on robots.txt: 200
curl -s -o /dev/null -w '%{http_code}\n' -A "compatible; GPTBot/1.4" https://www.example.com/
curl -s -o /dev/null -w '%{http_code}\n' -A "compatible; GPTBot/1.4" https://www.example.com/robots.txt
# Fake search crawler from your own machine: 403 with IP verification
curl -s -o /dev/null -w '%{http_code}\n' -A "compatible; OAI-SearchBot/1.4" https://www.example.com/
tail -f /var/log/apache2/ai-crawlers.log
```

These tests are only indicative: curl reproduces the User-Agent, not the crawler's address. In our tests, with the local address declared as an OpenAI range, `OAI-SearchBot` got a 200 and `Claude-SearchBot` a 403, as expected. The error log gives the reason for each denial (`AH01630: client denied by server configuration`).

## Pitfalls to avoid

- **The User-Agent alone proves nothing.** Denying by name only stops honest crawlers; only the comparison with IP ranges keeps impostors out, and only for vendors that publish them.
- **An overly broad pattern.** `bot` would deny Googlebot and Bingbot.
- **Blocking a vendor's IP ranges to refuse training.** Anthropic warns that blocking its IP addresses "may not work correctly or persistently guarantee an opt-out", because it also stops its crawlers from reading robots.txt.
- **Outdated lists.** Ranges change; a search crawler denied in the log often points to a list that is too old.

## Rolling back

In a `.htaccess`, delete or comment out the block you added; the effect is immediate. In the server configuration, remove the `<Location>` block and the `SetEnvIfNoCase` lines, then run `apachectl configtest` and `apachectl graceful`, and delete the cron job.

## Official sources

- Apache 2.4, mod_authz_core (Require, RequireAll, AuthMerging): <https://httpd.apache.org/docs/2.4/mod/mod_authz_core.html>
- Apache 2.4, mod_authz_host (Require ip): <https://httpd.apache.org/docs/2.4/mod/mod_authz_host.html>
- Apache 2.4, mod_setenvif: <https://httpd.apache.org/docs/2.4/mod/mod_setenvif.html>
- Apache 2.4, mod_rewrite: <https://httpd.apache.org/docs/2.4/mod/mod_rewrite.html>
- Apache 2.4, Access Control: <https://httpd.apache.org/docs/2.4/howto/access.html>
- Apache 2.4, Upgrading to 2.4 from 2.2: <https://httpd.apache.org/docs/2.4/upgrading.html>
- Apache 2.4, mod_log_config (CustomLog env=): <https://httpd.apache.org/docs/2.4/mod/mod_log_config.html>
- Apache 2.4, mod_headers: <https://httpd.apache.org/docs/2.4/mod/mod_headers.html>
- Apache 2.4, mod_remoteip: <https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html>
- Apache 2.4, mod_ratelimit: <https://httpd.apache.org/docs/2.4/mod/mod_ratelimit.html>
- RFC 9309, Robots Exclusion Protocol: <https://www.rfc-editor.org/rfc/rfc9309>
- OpenAI, Overview of OpenAI Crawlers: <https://developers.openai.com/api/docs/bots>
- Anthropic, Does Anthropic crawl data from the web: <https://support.claude.com/en/articles/8896518>
- Perplexity, Perplexity Crawlers: <https://docs.perplexity.ai/docs/resources/perplexity-crawlers>
