meta-externalfetcher
Meta's agent for fetching individual links at a user's request and supporting agentic AI features. Meta says it may bypass robots.txt rules for these fetches.
| Operator | Meta |
|---|---|
| Purpose | Browsing — Meta AI (user requests and agents) |
| robots.txt token | meta-externalfetcher |
| User-Agent | meta-externalfetcher/1.1 |
| Obeys robots.txt | Not always: may ignore it when a user triggers the visit (per the publisher) |
| Official IP ranges | Not published |
| Reverse DNS | Not documented |
| Official documentation | https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/ |
Live activity of meta-externalfetcher
No visit from meta-externalfetcher recorded on iabig.com over the last 30 days.
Last updated: 2026-10-25 12:07:01 UTC. See the Observatory
What meta-externalfetcher does
Meta-ExternalFetcher fetches individual links at a user's request. It also supports product functions such as evaluating and improving agentic AI capabilities, including helping AI navigate websites to complete tasks for users.
What Meta states:
- This crawler may bypass robots.txt because it performs fetches that users requested.
- The published user-agent string is
meta-externalfetcher/1.1, sometimes followed by a link to the documentation. - For Meta's other crawlers, blocking is done with a
Disallowrule in robots.txt.
Two points are worth noting. First, the description goes beyond fetching a link a user shared: it also covers evaluating and improving agentic capabilities, which is broader than answering a single question. Second, Meta publishes no IP list or verification method for this agent, so official documentation alone cannot tell a genuine Meta request from an imitation.
For a site that wants to refuse these visits, robots.txt is therefore not enough, and a server-side block can only rely on the user-agent string, which is trivial to fake.
Block or allow meta-externalfetcher
A robots.txt rule may not be honored, as Meta says explicitly. A server-side block on the user agent stops Meta AI features from reading your pages on users' behalf.
To block it, add this to your robots.txt:
User-agent: meta-externalfetcher
Disallow: /To allow it explicitly (copy any private rules you have):
User-agent: meta-externalfetcher
Allow: /A group naming meta-externalfetcher replaces the User-agent: * group for it. Build a complete robots.txt with the generator and check it with the tester.
Verify a meta-externalfetcher visit
Anyone can write “meta-externalfetcher” in a User-Agent. Paste the IP address and User-Agent from your logs into the IP verifier to know whether the visit is genuine.
Other Meta crawlers
- meta-externalagent — Training