# How to Block AI Crawlers on IIS - robots.txt, URL Rewrite, Request Filtering and IP Restrictions

> On IIS, publish robots.txt, deny AI training crawlers with a URL Rewrite rule in web.config and learn the limits of request filtering and IP restrictions.

Source: https://iabig.com/guides/platforms/iis-ai-crawlers/ — Guides CC BY 4.0, extraits de code MIT

On a Windows server running IIS, AI crawlers are handled with tools that were not designed for them: a static robots.txt file, the URL Rewrite module, Request Filtering and IP address restrictions. Each has its strengths and blind spots, and picking the wrong one can block robots.txt itself or slow down the whole site. This guide offers a configuration that refuses model training while letting the search crawlers of ChatGPT, Claude and Perplexity through, explains what each module can do according to Microsoft's documentation, and states plainly what IIS cannot do on its own. We were not able to run these configurations on an IIS server: they follow the official documentation and the `web.config` file was checked as valid XML, but test them on a staging site first.

## Publishing robots.txt

Put `robots.txt` in the site's physical folder (often under `C:\inetpub\wwwroot` for the Default Web Site). IIS serves it like any other static file. Check that it returns a 200, as plain text, and that no rule blocks it: a crawler that cannot read the file cannot discover your refusal in it, and RFC 9309 treats a 5xx error on this file as a full disallow.

robots.txt is what carries your policy to crawlers that respect it, which OpenAI, Anthropic and Perplexity state is the case for their crawlers. It is also the only way to refuse training to Google (`Google-Extended`) and Apple (`Applebot-Extended`), tokens that do not correspond to any separate crawler. The server rules that follow are only a complement, against crawlers that ignore the file.

## Denying by User-Agent with URL Rewrite

The URL Rewrite module is not part of the default IIS installation: it is installed separately from iis.net. Its "distributed" rules go in `web.config`, under `system.webServer/rewrite/rules`, and are evaluated in order. A condition can read any request header in the form `{HTTP_HEADER_NAME}`, so `{HTTP_USER_AGENT}` for the User-Agent.

```xml
<?xml version="1.0" encoding="UTF-8"?>
<!-- web.config at the site root - requires the URL Rewrite module -->
<configuration>
  <system.webServer>
    <rewrite>
      <rules>
        <!-- Deny AI training crawlers (adjust the list) -->
        <rule name="Deny AI training crawlers" stopProcessing="true">
          <!-- Every URL except robots.txt.
               At site level, the pattern receives the path without the leading slash. -->
          <match url="^robots\.txt$" negate="true" />
          <conditions logicalGrouping="MatchAll">
            <add input="{HTTP_USER_AGENT}"
                 pattern="GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent"
                 ignoreCase="true" />
          </conditions>
          <!-- 403 response; AbortRequest would drop the connection without a response -->
          <action type="CustomResponse" statusCode="403"
                  statusReason="Forbidden"
                  statusDescription="Access denied to AI training crawlers." />
        </rule>
      </rules>
    </rewrite>
  </system.webServer>
</configuration>
```

Several points in Microsoft's documentation explain this rule. The `<match>` pattern is compared with the path relative to the location of the `web.config`, without the query string: at the site root, `/robots.txt` arrives as `robots.txt`, hence the missing slash in the pattern. The `CustomResponse` action responds with the chosen status code and stops evaluation of subsequent rules. If your `web.config` already contains a `<rewrite>` block, do not add a second one; insert the rule at the top of the existing list.

One side effect is worth knowing before going to production: according to the documentation, a rule that uses a server variable missing from a specific list (`HTTP_USER_AGENT` is not on it) makes the whole rule set "unsafe" for output caching. The module then disables kernel-mode caching for all requests, rewritten or not. On a high-traffic site that relies on this cache, measure the impact.

## Request Filtering: possible, but less suitable

Request Filtering, installed by default, can also inspect a header thanks to filtering rules (`filteringRules`, available since IIS 7.5):

```xml
<!-- Inside <system.webServer><security> in web.config -->
<requestFiltering>
  <filteringRules>
    <filteringRule name="Deny AI training crawlers"
                   scanUrl="false" scanQueryString="false" scanAllRaw="false">
      <scanHeaders>
        <add requestHeader="User-Agent" />
      </scanHeaders>
      <!-- Empty appliesTo: the rule applies to every request -->
      <denyStrings>
        <add string="GPTBot" />
        <add string="ClaudeBot" />
        <add string="CCBot" />
      </denyStrings>
    </filteringRule>
  </filteringRules>
</requestFiltering>
```

Two limitations make us prefer URL Rewrite. A blocked request receives a 404 error, logged with substatus 404.19 ("Denied by filtering rule"), so the crawler does not understand that it has been refused. More importantly, a rule can only be restricted by file extension (`appliesTo`), not by path: without care, it also blocks `robots.txt`. The documentation also does not say whether string matching is case-sensitive; check this on your server before relying on it.

## IP address restrictions

The **IP and Domain Restrictions** module is not installed by default: it is added as a role service (**Web Server** > **Security** > **IP and Domain Restrictions**). It denies or allows addresses or ranges, expressed with a subnet mask rather than in CIDR notation. The documentation for the `<ipSecurity>` element only mentions IPv4. Microsoft configures this module in `applicationHost.config`: with AppCmd, you need to add `/commit:apphost`, or go through **IP Address and Domain Restrictions** in IIS Manager.

```text
:: Command Prompt, run as administrator
:: Deny an abusive range (example: 203.0.113.0/24 = mask 255.255.255.0)
%windir%\system32\inetsrv\appcmd.exe set config "Default Web Site" -section:system.webServer/security/ipSecurity /+"[ipAddress='203.0.113.0',subnetMask='255.255.255.0',allowed='False']" /commit:apphost
```

This module cannot combine an address with a User-Agent. With its built-in modules alone, IIS therefore cannot reproduce the check described in our [nginx](/guides/platforms/nginx-ai-crawlers/) and [Apache](/guides/platforms/apache-ai-crawlers/) guides, which only accepts a search crawler if it comes from the ranges published by its vendor: URL Rewrite compares regular expressions, not CIDR ranges. If that check matters to you, put it upstream, on a CDN, a web application firewall or a reverse proxy.

For rate limiting, the same module has offered dynamic restrictions (`dynamicIpSecurity`) since IIS 8.0: denying an address beyond a number of concurrent requests or a number of requests over a period of time. They apply to all visitors, crawlers or not. Enable `enableLoggingOnlyMode` first, which logs denials without enforcing them.

## X-Robots-Tag

```xml
<!-- In web.config: header on the whole /documents folder -->
<location path="documents">
  <system.webServer>
    <httpProtocol>
      <customHeaders>
        <add name="X-Robots-Tag" value="noindex" />
      </customHeaders>
    </httpProtocol>
  </system.webServer>
</location>
```

Google documents this header; the OpenAI, Anthropic and Perplexity pages we consulted for this guide do not mention it. For those crawlers, robots.txt remains the only documented control.

## Testing

In Windows PowerShell 5.1, `curl` is an alias for `Invoke-WebRequest`: call `curl.exe` explicitly, which ships with recent versions of Windows.

```powershell
# PowerShell. Training crawler: 403 expected
curl.exe -s -o NUL -w "%{http_code}\n" -A "compatible; GPTBot/1.4" https://www.example.com/
# Same crawler on robots.txt: 200 expected
curl.exe -s -o NUL -w "%{http_code}\n" -A "compatible; GPTBot/1.4" https://www.example.com/robots.txt
# Search crawler: 200 expected
curl.exe -s -o NUL -w "%{http_code}\n" -A "compatible; OAI-SearchBot/1.4" https://www.example.com/
```

These tests are only indicative: curl reproduces the User-Agent, not the crawler's address. W3C logs are stored by default under `%SystemDrive%\inetpub\logs\LogFiles`, in one subfolder per site, and the User-Agent is among the fields logged by default; in these files, spaces in the User-Agent are replaced with `+`. To find denials:

```powershell
Select-String -Path "$env:SystemDrive\inetpub\logs\LogFiles\W3SVC1\*.log" -Pattern "GPTBot|ClaudeBot|CCBot" |
  Select-String -Pattern " 403 | 404 19 "
```

To write a robots.txt without token mistakes, you can start from the [AI robots.txt generator](/tools/robots-txt-generator/), or copy the ready-made file on the [AI blocklists page](/tools/ai-blocklists/), regenerated every hour. That page has no IIS-specific format, but its list of user-agent tokens can go straight into the `pattern` of the URL Rewrite rule above, separated by `|`.

## Pitfalls to avoid

- **The User-Agent alone proves nothing.** A rule based on the name only stops honest crawlers, which robots.txt already stopped; a scraper posing as a browser gets through.
- **An overly broad pattern.** `bot` would deny Googlebot and Bingbot and remove you from search engines.
- **Blocking robots.txt by accident**, especially with Request Filtering.
- **Blocking a vendor's IP ranges to refuse training.** Anthropic warns that blocking its IP addresses "may not work correctly or persistently guarantee an opt-out", because it also stops its crawlers from reading robots.txt.
- **Forgetting the proxy.** Behind a load balancer, IIS sees the intermediary's address; the IP restrictions module offers a proxy mode (`enableProxyMode`) that reads the `x-forwarded-for` header.

## Rolling back

Delete the rule from `web.config` (or from **URL Rewrite** in IIS Manager), remove the rule from the **Rules** tab of **Request Filtering**, and delete the entries added in **IP Address and Domain Restrictions**. Keep a copy of the file before every change: it is the fastest way back.

## Official sources

- Microsoft, URL Rewrite Module Configuration Reference: <https://learn.microsoft.com/en-us/iis/extensions/url-rewrite-module/url-rewrite-module-configuration-reference>
- Microsoft, URL Rewrite (download): <https://www.iis.net/downloads/microsoft/url-rewrite>
- Microsoft, Request Filtering Rule (filteringRule): <https://learn.microsoft.com/en-us/iis/configuration/system.webserver/security/requestfiltering/filteringrules/filteringrule/>
- Microsoft, IP Security (ipSecurity): <https://learn.microsoft.com/en-us/iis/configuration/system.webserver/security/ipsecurity/>
- Microsoft, Adding IP Security (add): <https://learn.microsoft.com/en-us/iis/configuration/system.webserver/security/ipsecurity/add>
- Microsoft, Dynamic IP Security: <https://learn.microsoft.com/en-us/iis/configuration/system.webserver/security/dynamicipsecurity/>
- Microsoft, Custom Headers: <https://learn.microsoft.com/en-us/iis/configuration/system.webserver/httpprotocol/customheaders/>
- Microsoft, Default Log File Settings (logFile): <https://learn.microsoft.com/en-us/iis/configuration/system.applicationhost/sites/sitedefaults/logfile/>
- RFC 9309, Robots Exclusion Protocol: <https://www.rfc-editor.org/rfc/rfc9309>
- OpenAI, Overview of OpenAI Crawlers: <https://developers.openai.com/api/docs/bots>
- Anthropic, Does Anthropic crawl data from the web: <https://support.claude.com/en/articles/8896518>
- Perplexity, Perplexity Crawlers: <https://docs.perplexity.ai/docs/resources/perplexity-crawlers>
- Google, Robots meta tag and X-Robots-Tag: <https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag>
