Search crawlers (cited in answers)
| Crawler | Purpose | Visits (30 d) |
|---|---|---|
| OAI-SearchBot OpenAI | OpenAI's crawler for surfacing websites in ChatGPT's search features. Sites that block it are left out of ChatGPT search answers, though they can still appear as navigational links. | — |
| Claude-SearchBot Anthropic | Anthropic's crawler for improving the relevance and accuracy of Claude's search responses. Blocking it may reduce how visible a site is in those results. | — |
| PerplexityBot Perplexity | Perplexity's crawler for surfacing and linking websites in its search results. Perplexity says it is not used to crawl content for AI foundation models. | — |
| Googlebot | Google's main crawler, in smartphone and desktop variants. It feeds Google Search, including Discover and all Search features, as well as Google Images, Video mode and Google News. | 4675 |
| Bingbot Microsoft | Bing's standard crawler. According to Microsoft's webmaster guidelines, the index it builds serves Bing search, Copilot and grounding results delivered through APIs. | 8 |
| Applebot Apple | Apple's crawler behind search in Spotlight, Siri and Safari. Its data may also train Apple's foundation models unless publishers opt out through Applebot-Extended. | — |
| DuckAssistBot DuckDuckGo | DuckDuckGo's crawler that fetches pages in real time for its AI-assisted answers, which cite their sources. DuckDuckGo says the data is not used to train AI models. | — |
Browsing agents (on a user's request)
| Crawler | Purpose | Visits (30 d) |
|---|---|---|
| ChatGPT-User OpenAI | OpenAI's agent for visiting a page when someone asks ChatGPT or a custom GPT a question. OpenAI says robots.txt rules may not apply to these user-initiated requests. | — |
| Claude-User Anthropic | Anthropic's agent for fetching a site when a Claude user asks a question that needs it. Anthropic presents robots.txt as the way to control these user-initiated requests. | — |
| Perplexity-User Perplexity | Perplexity's agent for visiting a page to answer a user's question and cite it. Perplexity says that, because a user requested the fetch, it generally ignores robots.txt. | — |
| meta-externalfetcher Meta | Meta's agent for fetching individual links at a user's request and supporting agentic AI features. Meta says it may bypass robots.txt rules for these fetches. | — |
| MistralAI-User Mistral AI | Mistral AI's agent for visiting a page when a user asks Vibe a question, with a link to the source. It does not crawl automatically or collect training data. | — |
Model training crawlers
| Crawler | Purpose | Visits (30 d) |
|---|---|---|
| GPTBot OpenAI | OpenAI's crawler for web content that may be used to train its generative AI foundation models. Its robots.txt setting is independent of the one that governs ChatGPT search. | — |
| ClaudeBot Anthropic | Anthropic's crawler for web content that could contribute to training its generative AI models. Anthropic says it honors robots.txt and the non-standard Crawl-delay directive. | 172 |
| Google-Extended | A Google robots.txt token with no crawler of its own. It controls whether crawled content may be used to train Gemini models and for grounding in Gemini Apps and Vertex AI. | — |
| Applebot-Extended Apple | An Apple robots.txt token with no crawler of its own. It lets publishers refuse the use of Applebot data to train Apple's foundation models, without affecting search. | — |
| meta-externalagent Meta | Meta's crawler for use cases such as training foundation AI models or improving products by indexing content directly. Meta publishes no IP list for it. | — |
| Amazonbot Amazon | Amazon's crawler for improving its products and services. Amazon says the data may be used to train its AI models. It honors robots.txt but not Crawl-delay. | — |
| CCBot Common Crawl | The crawler of Common Crawl, a non-profit that maintains an open archive of the web that researchers, companies and individuals can download for free. | — |
| Bytespider ByteDance | A crawler attributed to ByteDance that shows up widely in logs and blocklists. No official documentation of its purpose, robots.txt handling or IP addresses could be found. | — |
Last updated: 2026-10-25 14:07:01 UTC. Verified visits to iabig.com, see the Observatory. To build your robots.txt: robots.txt generator.