# How to Track AI Crawler Traffic in nginx and Apache Logs - grep, awk and GoAccess

> Count visits from GPTBot, ClaudeBot, PerplexityBot and other AI crawlers in nginx or Apache logs, per crawler and per page, with a complete script.

Source: https://iabig.com/guides/ai-crawlers/ai-crawler-traffic-server-logs/ — Guides CC BY 4.0, extraits de code MIT

Analytics tools built on a JavaScript snippet only see visitors that run the snippet. AI crawlers that do not execute it, such as CCBot according to Common Crawl, never show up there. To know who actually reads your site, which pages and how often, the reliable source is still the web server's access log. This guide starts from the standard log format, gives the useful grep, awk and GoAccess commands, then a complete script that produces a per-crawler summary.

## The log format you need to know

nginx uses the "combined" format by default, and Apache provides it under the same name. A line looks like this:

```text
20.171.206.5 - - [24/Oct/2026:10:00:01 +0200] "GET /guide/ HTTP/1.1" 200 5120 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
```

In nginx, this format is predefined under the name `combined`, and it is the one `access_log` uses when no format is specified:

```nginx
# Built into nginx (no need to copy it)
log_format combined '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';
```

In Apache, the format has the same name. Make sure your `CustomLog` actually uses it, and not `common`, which does not record the User-Agent:

```apache
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined
CustomLog ${APACHE_LOG_DIR}/access.log combined
```

Usual locations are `/var/log/nginx/access.log` for nginx, `/var/log/apache2/access.log` on Debian and Ubuntu, and `/var/log/httpd/access_log` on Red Hat-based distributions. Older logs are often compressed (`access.log.2.gz`), and `zcat -f` reads compressed and uncompressed files alike.

Splitting on the double quote is the key to the awk commands that follow. With `awk -F'"'`, field `$2` holds the request (`GET /guide/ HTTP/1.1`), `$3` the status code and size, and `$6` the User-Agent.

## Crawlers to look for

The tokens below are the ones published by the operators. They appear in the User-Agent header of requests, except for `Google-Extended` and `Applebot-Extended`, which are only robots.txt tokens. You will never see them in a log.

| Family | Tokens to search for |
|---|---|
| Training | `GPTBot`, `ClaudeBot`, `CCBot`, `meta-externalagent`, `MistralAI-Training`, `Amazonbot`, `Bytespider` |
| Search and indexing | `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`, `Applebot`, `Amzn-SearchBot`, `meta-webindexer`, `MistralAI-Index`, `DuckAssistBot` |
| User-initiated | `ChatGPT-User`, `Claude-User`, `Perplexity-User`, `MistralAI-User`, `meta-externalfetcher`, `Amzn-User`, `Google-Agent` |

Version numbers change (`GPTBot/1.4` today), so always search for the token, never the full string.

## Quick commands

```bash
LOG=/var/log/nginx/access.log
BOTS='GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|meta-externalagent|meta-externalfetcher|meta-webindexer|Applebot|Amazonbot|Amzn-SearchBot|Amzn-User|MistralAI-User|MistralAI-Index|MistralAI-Training|DuckAssistBot|Google-Agent'

# Total number of AI crawler requests
grep -ciE "$BOTS" "$LOG"

# Requests per crawler, archived logs included
# (grep -o would also count the documentation URLs inside the User-Agent,
#  so only the first match in the User-Agent field is kept; requires GNU awk)
zcat -f /var/log/nginx/access.log* \
  | awk -F'"' -v bots="$BOTS" 'BEGIN { IGNORECASE = 1 } match($6, bots) { print substr($6, RSTART, RLENGTH) }' \
  | sort | uniq -c | sort -rn

# Top 20 pages read by GPTBot (URL parameters stripped)
awk -F'"' 'tolower($6) ~ /gptbot/ { split($2, r, " "); sub(/\?.*/, "", r[2]); print r[2] }' "$LOG" \
  | sort | uniq -c | sort -rn | head -20

# Status codes returned to ClaudeBot (spot 403, 404 and 5xx)
awk -F'"' '$6 ~ /ClaudeBot/ { split($3, s, " "); print s[1] }' "$LOG" | sort | uniq -c | sort -rn

# Requests per day for OAI-SearchBot
awk -F'"' '$6 ~ /OAI-SearchBot/ { print substr($1, index($1, "[") + 1, 11) }' "$LOG" | uniq -c

# Who reads robots.txt, and with which status code
awk -F'"' '$2 ~ /^GET \/robots\.txt/ { split($3, s, " "); print s[1], $6 }' "$LOG" \
  | grep -iE "$BOTS" | sort | uniq -c | sort -rn

# Bytes transferred per crawler
awk -F'"' -v bots="$BOTS" 'BEGIN { IGNORECASE = 1 }
  match($6, bots) { split($3, s, " "); o[substr($6, RSTART, RLENGTH)] += s[2] }
  END { for (b in o) printf "%12d  %s\n", o[b], b }' "$LOG" | sort -rn
```

The commands that use `IGNORECASE` require GNU awk (`gawk`). With another awk, remove the `BEGIN` block and match the tokens' case exactly.

The robots.txt command is useful for checking your setup. Under RFC 9309, a crawler that gets a 5xx error on that file must treat the whole site as disallowed.

## Writing AI crawlers to a dedicated log

Rather than filtering after the fact, you can have the server write these requests to a separate file.

With nginx, in the `http` block:

```nginx
map $http_user_agent $ai_bot {
    default 0;
    "~*(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|meta-external|meta-webindexer|Amazonbot|Amzn-|MistralAI-|DuckAssistBot|Google-Agent|Applebot)" 1;
}
```

Then in the `server` block:

```nginx
# access_log in server replaces the inherited directives, so the main log is kept.
access_log /var/log/nginx/access.log combined;
access_log /var/log/nginx/ai-bots.log combined if=$ai_bot;
```

With Apache (`mod_setenvif` and `mod_log_config` modules):

```apache
SetEnvIfNoCase User-Agent "(GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|meta-external|meta-webindexer|Amazonbot|Amzn-|MistralAI-|DuckAssistBot|Google-Agent|Applebot)" ai_bot
CustomLog ${APACHE_LOG_DIR}/ai-bots.log combined env=ai_bot
```

Remember to add this file to your `logrotate` configuration, like the server's other logs.

## GoAccess

GoAccess parses logs in the combined format and produces an HTML report. Since version 1.12 (September 2026), it puts AI crawlers in a separate "AI Crawlers" category, distinct from traditional search engines.

```bash
# Full report
goaccess /var/log/nginx/access.log --log-format=COMBINED -o report.html

# Report limited to AI crawlers, compressed logs included
zcat -f /var/log/nginx/access.log* \
  | grep -iE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot|Bytespider|meta-external' \
  | goaccess - --log-format=COMBINED -o ai-bots.html
```

With an older version, declare the tokens yourself in a list passed with `--browsers-file` (one word, one or more tabs, then the category):

```bash
printf '%s\t\tCrawlers\n' GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User > ai.list
goaccess /var/log/nginx/access.log --log-format=COMBINED --browsers-file=ai.list -o report.html
```

If your server writes the virtual host at the start of each line, use `VCOMBINED` instead of `COMBINED`.

## Ready-to-use script

This script reads one or more logs in the combined format, compressed or not, and prints the number of requests per crawler, status codes, the number of distinct IP addresses, activity per day and the most-read pages.

```bash
#!/usr/bin/env bash
# ai-bot-logs.sh - summary of AI crawler visits ("combined" format)
# Usage: ./ai-bot-logs.sh /var/log/nginx/access.log*   (.gz files accepted)
set -euo pipefail
[ "$#" -ge 1 ] || { echo "usage: $0 log_file [log_file...]" >&2; exit 1; }

# Tokens searched for in the User-Agent (case-insensitive)
BOTS="GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User Google-Agent Applebot Amazonbot Amzn-SearchBot Amzn-User meta-externalagent meta-externalfetcher meta-webindexer CCBot Bytespider MistralAI-User MistralAI-Index MistralAI-Training DuckAssistBot"

zcat -f -- "$@" | awk -F'"' -v bots="$BOTS" '
BEGIN { n = split(bots, B, " "); for (i = 1; i <= n; i++) L[i] = tolower(B[i]) }
{
  ua = tolower($6); bot = ""
  for (i = 1; i <= n; i++) if (index(ua, L[i])) { bot = B[i]; break }
  if (bot == "") next
  split($2, req, " "); url = req[2]; sub(/\?.*/, "", url)      # path without parameters
  split($3, st, " "); code = st[1]
  split($1, f, "["); day = substr(f[2], 1, 11)                  # 25/Oct/2026
  split($1, h, " "); ip = h[1]
  hits[bot]++; codes[bot " " code]++; pages[bot "\t" url]++
  days[day " " bot]++; ips[bot " " ip] = 1
}
END {
  print "== Requests per crawler"
  for (b in hits) printf "%8d  %s\n", hits[b], b | "sort -rn"; close("sort -rn")
  print "\n== Status codes per crawler"
  for (k in codes) printf "%8d  %s\n", codes[k], k | "sort -k2,2 -k1,1rn"; close("sort -k2,2 -k1,1rn")
  print "\n== Distinct IP addresses per crawler"
  for (k in ips) { split(k, a, " "); nip[a[1]]++ }
  for (b in nip) printf "%8d  %s\n", nip[b], b | "sort -rn"; close("sort -rn")
  print "\n== Requests per day"
  for (k in days) printf "%8d  %s\n", days[k], k | "sort -k2,2 -k1,1rn"; close("sort -k2,2 -k1,1rn")
  print "\n== Top 20 pages (all AI crawlers)"
  for (k in pages) { split(k, a, "\t"); tot[a[2]] += pages[k] }
  for (u in tot) printf "%8d  %s\n", tot[u], u | "sort -rn | head -20"; close("sort -rn | head -20")
}'
```

```bash
chmod +x ai-bot-logs.sh
./ai-bot-logs.sh /var/log/nginx/access.log /var/log/nginx/access.log.*.gz
sudo ./ai-bot-logs.sh /var/log/apache2/access.log*
```

Sample output on a small test log:

```text
== Requests per crawler
       3  GPTBot
       1  PerplexityBot
       1  ClaudeBot
       1  Claude-SearchBot

== Status codes per crawler
       1  Claude-SearchBot 200
       1  ClaudeBot 200
       1  GPTBot 200
       1  GPTBot 403
       1  GPTBot 404
       1  PerplexityBot 200
```

The script counts what the requests **claim** to be. Dates are sorted as text, which works for a single month. For a longer period, export the data to a spreadsheet or a database.

## Breaking results down crawler by crawler

The overall summary mixes crawlers with very different roles. To compare what each crawler reads, a loop is enough. It produces a page ranking per crawler, which shows whether a search crawler actually reaches your important content or wears itself out on filter, pagination or archive pages.

```bash
for bot in GPTBot OAI-SearchBot ClaudeBot Claude-SearchBot PerplexityBot; do
  echo "== $bot"
  zcat -f /var/log/nginx/access.log* \
    | awk -F'"' -v b="$bot" 'index(tolower($6), tolower(b)) { split($2, r, " "); sub(/\?.*/, "", r[2]); print r[2] }' \
    | sort | uniq -c | sort -rn | head -10
done
```

## Tracking trends over time

Logs are often deleted after a few weeks by `logrotate`. To keep a record, save a per-crawler count to a CSV file every night, from the previous day's log (here `access.log.1`, the usual name after daily rotation):

```bash
# /etc/cron.d/ai-bots - every day at 00:30
30 0 * * * root gawk -F'"' 'BEGIN { IGNORECASE = 1; d = strftime("\%Y-\%m-\%d", systime() - 86400) } match($6, /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|CCBot/) { n[substr($6, RSTART, RLENGTH)]++ } END { for (b in n) print d ";" b ";" n[b] }' /var/log/nginx/access.log.1 >> /var/log/ai-bots.csv
```

In a cron file, the `%` character must be escaped, hence the `\%`. The `strftime` and `systime` functions and `IGNORECASE` are specific to GNU awk, which is why the line calls `gawk` explicitly (on Debian and Ubuntu, the default `awk` is often `mawk`). The resulting file opens in any spreadsheet, with one column for the date, one for the crawler and one for the request count.

What should you look at first? Three signals are usually enough. Search crawlers (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`) should show up regularly if you allow them. If they stay absent, check robots.txt and the firewall. User-initiated agents (`ChatGPT-User`, `Claude-User`, `Perplexity-User`) show which pages are read at the moment a user asks a question. Finally, a crawler you have disallowed in robots.txt that keeps reading pages other than `/robots.txt` needs checking. Either it is an impostor, or the rule does not target it correctly.

## Read the numbers with caution

- **Spoofing**: a User-Agent is easy to fake. An IP that claims to be GPTBot but is not in OpenAI's ranges is not GPTBot. Cross-check the addresses against the official JSON files, as explained in [How to verify GPTBot, ClaudeBot and PerplexityBot](/guides/ai-crawlers/verify-gptbot-claudebot-perplexitybot/).
- **CDNs and proxies**: behind a CDN, the first column holds the proxy's address. Enable nginx's `realip` module or Apache's `mod_remoteip` to log the client IP, or use the CDN's logs.
- **Shared crawls**: OpenAI says a single crawl can serve both `GPTBot` and `OAI-SearchBot` when both are allowed. The absence of one does not mean your pages are being ignored.
- **On-demand visits**: a `ChatGPT-User`, `Claude-User` or `Perplexity-User` request usually corresponds to a user's question. It is the closest indicator of an assistant actually reading your content, but it says nothing about how many people will see the answer.
- **Error codes**: a high rate of 403 or 429 for a search crawler often means a firewall or rate limit is blocking it without your knowledge.

To check what your robots.txt and pages tell these crawlers, use the [IABIG checker](/).

## Official sources

- nginx, logging module: <https://nginx.org/en/docs/http/ngx_http_log_module.html>
- nginx, map module: <https://nginx.org/en/docs/http/ngx_http_map_module.html>
- Apache, mod_log_config: <https://httpd.apache.org/docs/2.4/mod/mod_log_config.html>
- Apache, mod_setenvif: <https://httpd.apache.org/docs/2.4/mod/mod_setenvif.html>
- GoAccess, manual and 1.12 release notes: <https://goaccess.io/man> and <https://github.com/allinurl/goaccess/blob/master/ChangeLog>
- OpenAI, Overview of OpenAI Crawlers: <https://developers.openai.com/api/docs/bots>
- Anthropic, Does Anthropic crawl data from the web: <https://support.claude.com/en/articles/8896518>
- Perplexity, Perplexity Crawlers: <https://docs.perplexity.ai/docs/resources/perplexity-crawlers>
- Common Crawl, FAQ (JavaScript execution): <https://commoncrawl.org/faq>
- Google, Google-Extended: <https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers>
- Google, user-triggered fetchers (Google-Agent): <https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers>
- Apple, About Applebot: <https://support.apple.com/en-us/119829>
