- How to Block AI Crawlers on IIS - robots.txt, URL Rewrite, Request Filtering and IP Restrictions
On IIS, publish robots.txt, deny AI training crawlers with a URL Rewrite rule in web.config and learn the limits of request filtering and IP restrictions.
- How to Block AI Crawlers in nginx - User-Agent Rules, Official IP Ranges and Rate Limits
A tested nginx configuration that denies AI training crawlers, lets genuine AI search crawlers through, rate-limits them and writes them to a dedicated log.
- How to Block AI Crawlers in Apache 2.4 - robots.txt, .htaccess, Require not env and IP Ranges
Deny AI training crawlers in Apache 2.4 with .htaccess, mod_setenvif or mod_rewrite, and only accept AI search crawlers coming from their official IP ranges.
- Block or Allow AI Crawlers on WordPress - Virtual robots.txt, Physical File and PHP Filter
Where to edit robots.txt on WordPress to block GPTBot or ClaudeBot yet stay visible in ChatGPT Search, Claude and Perplexity, by file, PHP filter or plugin.
- Block or Allow AI Crawlers on Wix - The robots.txt Editor and Its Limits
Edit robots.txt on Wix to block GPTBot or ClaudeBot without blocking ChatGPT Search, Claude and Perplexity, where to find the editor and the pitfalls to avoid.
- Block or Allow AI Crawlers on Webflow - robots.txt Rules and the webflow.io Subdomain
Write Webflow robots.txt rules that block GPTBot or ClaudeBot without blocking ChatGPT, Claude and Perplexity, and avoid the trap of the AI bots toggle.
- Block or Allow AI Crawlers on Squarespace - What the Crawlers Setting Really Does
Squarespace doesn't let you edit robots.txt. What its AI crawler checkbox blocks, what it lets through, and how to check the file that is actually served.
- Block or Allow AI Crawlers on Shopify - Customize robots.txt.liquid
Add groups to robots.txt.liquid that block GPTBot or ClaudeBot while keeping your store visible in ChatGPT, Claude and Perplexity, plus the limits to know.
- AI Crawlers on Netlify - robots.txt, the _headers File and an Edge Function Filter
Publish robots.txt on Netlify, add X-Robots-Tag with _headers and filter AI crawlers by User-Agent with an Edge Function or the User Agent Blocker extension.
- AI Crawlers on Next.js and Vercel - robots.ts, X-Robots-Tag and the Vercel Firewall
Configure robots.txt with app/robots.ts in Next.js, set an X-Robots-Tag header and tune the Vercel Firewall without blocking ChatGPT Search or Perplexity.