robots.txt generator for AI crawlers

Your robots.txt

Copy Test this robots.txt →

How to use it

1. Pick a preset, then fine-tune crawler by crawler if needed.
2. Save the output as robots.txt at the root of your site (https://yoursite.com/robots.txt). WordPress, Shopify, Wix and Webflow each have their own way to edit it.
3. Check the result with the AI visibility checker.

Training, search, browsing: what's the difference?

AI companies run separate crawlers for separate purposes. Training crawlers (GPTBot, ClaudeBot, Google-Extended…) collect pages to train models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot…) index pages so they can be cited in answers. Browsing agents (ChatGPT-User, Claude-User, Perplexity-User…) fetch a page when a user asks for it.

The recommended preset blocks training and allows search and browsing: your content is not used to train models, but your site can still be cited, with a link, in ChatGPT, Claude or Perplexity. Blocking Googlebot or Bingbot would remove your site from classic search engines, so they stay allowed by default.

robots.txt is a request, not a lock: compliant crawlers follow it, others ignore it. Some user-triggered browsing agents may also ignore it, according to their own publishers. See each crawler's page for details.

More free tools