What Is Meta-ExternalAgent? What Meta's Crawlers Do, and Which to Block

What is meta-externalagent?

Meta-ExternalAgent is a web crawler run by Meta, the company behind Facebook, Instagram and Meta AI. Meta's crawler documentation says it "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly." In plain terms, it collects pages that may be used to train Meta's AI models. Meta's doc tells you to block it with robots.txt.

Most people meet it in a server log or a hosting dashboard, as a user agent they do not recognize. Meta's doc says the string will be similar to meta-externalagent/1.1, either alone or followed by a link to the doc in parentheses (read on Oct 31, 2026). Match on the name meta-externalagent rather than the full string, since the version number and link can change. At least one AI crawler directory lists it as version 1.0 followed by facebook.com/externalhit_uatext.php. That link belongs to facebookexternalhit, Meta's link-preview crawler, and Meta's doc shows version 1.1. Some also say it trains Llama. Meta's doc says "foundation AI models" and names no model.

What do Meta's five crawlers each do?

Meta lists five crawlers, and each does a different job. Only one of them is for AI training. The table sums up Meta's own descriptions from its Meta Web Crawlers doc, last updated May 21, 2026, which I read on Oct 31, 2026. The doc lists no crawler beyond these five, though Meta describes them as its "most common" crawlers, not a complete list.

The split matters because the crawlers serve different interests of yours. Meta-ExternalAgent takes content for training. Meta-WebIndexer is the one that can send people back to you, through citations in Meta AI answers. And facebookexternalhit makes your links look right when someone shares them. A site can block the first and keep the other two.

Businesses on Meta have one more document to read. Meta's updated Commercial Terms take effect on November 20, 2026, and cover business use of Meta, such as running ads, managing a Page, listing products or building apps. They authorize Meta "to use automated software (such as web scrapers, bots, robots, spiders, crawlers, and user agents)" on domains "associated with your products or services," for its advertising and business products and for the "integrity and security" of its products. The new text does not mention robots.txt or name a user agent, and the current terms have no such clause (read Oct 31, 2026). If you use Meta for business, read those terms before relying on a robots.txt block of meta-externalads.

What is the difference between meta-externalagent and facebookexternalhit?

Meta-ExternalAgent crawls the web on its own for uses such as AI training. Facebookexternalhit fetches a page mainly when someone shares its link on Facebook, Instagram or Messenger, to build the preview card with its title, description and image. It also refreshes that data and runs security checks. They are separate user agents, so a robots.txt group for one does not touch the other.

• Trigger: meta-externalagent crawls on Meta's schedule. Facebookexternalhit is triggered by a shared link, and Meta says the link-preview crawler refreshes link metadata "automatically every 30 days."
• Purpose: training and product indexing, against link previews.
• robots.txt: Meta's doc names meta-externalagent as blockable. It says facebookexternalhit "might bypass robots.txt" for security or integrity checks.
• Fixing a stale preview: Meta points to its Sharing Debugger, which forces a fresh crawl by facebookexternalhit. Nothing equivalent exists for meta-externalagent.

One detail explains the mix-ups in some bot directories. The link inside facebookexternalhit's user agent, facebook.com/externalhit_uatext.php, now redirects to the same Meta Web Crawlers doc that describes all five crawlers (checked Oct 31, 2026). Meta keeps one page for all of them, but each crawler still sends its own name.

How do Meta's crawlers compare with other AI companies' crawlers?

Meta follows the same pattern as OpenAI and Anthropic: one named crawler for training, one for AI search citations, and one for fetches a user asks for. Google and Apple work differently. Their training opt-outs, Google-Extended and Applebot-Extended, are tokens you name in robots.txt, not crawlers that visit your site. Google-Extended also controls whether Gemini Apps and Vertex AI use your pages to ground answers. I read each company's own crawler page on Oct 31, 2026 for this table.

Two differences matter for Meta. First, its training crawler has a wider brief than GPTBot or ClaudeBot. Meta's description adds "improving products by indexing content directly," so a block covers that use too. Second, Meta joins OpenAI, Google and Perplexity in saying its user-triggered fetcher may not follow robots.txt. Anthropic is the only company in the table that says all its bots honor it. For the full list of crawlers by company, the guide to AI crawlers goes through each one.

Should you block meta-externalagent?

Block meta-externalagent if you do not want your content used to train Meta's AI models. Keep it if you are happy for it to be. Meta's doc does not say that blocking it costs you anything in Meta AI answers or link previews, because separate crawlers handle those. So the decision is about training use, not about being found.

For most small-business sites, the stakes are low either way. Blocking also works only from now on: Meta's doc does not say that a block removes content already collected (checked Oct 31, 2026). Publishers weigh it differently, because their articles are the product. That shows in the survey below. If you want the general case for search versus training crawlers across OpenAI, Anthropic, Google and others, the guide to AI crawlers covers it.

This site, jnabear.com, does not name any of Meta's crawlers in its robots.txt. They fall under its * group, which allows everything except one internal path. That is a default, not a choice made for Meta: the file names the AI crawlers it allows on purpose, and Meta's are not among them. The goal here is to be cited, so I am leaving it that way for now.

How do you block meta-externalagent in robots.txt?

Add a group for it to your robots.txt file, at the root of your site. This blocks meta-externalagent and leaves the crawlers behind Meta AI citations and link previews unblocked:

User-agent: meta-externalagent
Disallow: /

Those two lines are enough. Meta-WebIndexer and facebookexternalhit are not named, so they keep following your * group. Write the name on its own, with no version number or slash: RFC 9309 says a product token may contain only letters, underscores and hyphens. And skip NoAI meta tags. Meta's doc says it uses "industry-standard practices like robots.txt rather than non-standard formats like NoAI tags."

Pick the rule by what you want to keep. Each row adds groups to your robots.txt and leaves the rest of the file as it is.

Leave facebookexternalhit off every list unless you want shared links to show no preview. Blocking meta-externalfetcher is a request Meta says it may not honor, since those fetches come from a user.

Two details trip people up. First, Meta says its crawlers "may cache the contents of robots.txt for up to 24 hours," so allow a day before you judge the result. Second, a crawler obeys only the groups that name it, and falls back to * only when none does. Under RFC 9309, the robots.txt standard, a named group does not inherit the rules in your * group. If you name meta-externalagent to block one folder, list every path you want blocked in that group. To test the result, paste your domain into the AI Crawler Checker, which I built. It reads your live robots.txt with the same RFC 9309 rules and shows the line that decided each crawler it checks, including meta-externalagent, meta-webindexer and meta-externalfetcher.

Which big sites block meta-externalagent?

Of 37 big sites whose robots.txt I read on Oct 31, 2026, 20 block meta-externalagent from their home page. 16 of them block it by name. Four block it only because their * group disallows the home page. Eleven of the 20 still allow meta-webindexer, so they decline training but stay open to citations in Meta AI.

How the sample was made: I chose 41 well-known news, reference, retail and website-builder sites before running the check, and did not add or drop any after seeing results. Four files could not be read: npr.org did not answer, politico.com and stackoverflow.com returned 403, and arstechnica.com returned 405. Those servers may answer real crawlers differently, so they are left out rather than counted as open. Each file was parsed with the AI Crawler Checker's own parser, for the home page (/) only. This is a convenience sample of large sites, not a measure of the web. It shows what robots.txt says, not what any firewall does.

These are the 20 sites that block at least one of Meta's crawlers. "Named" means the site wrote a group for that crawler. "By *" means the crawler fell under a catch-all group that disallows the home page. Squarespace.com names meta-externalagent in a group it shares with *, so it gets the same rules as every other crawler.

The other 17 readable sites allowed all five crawlers on their home page: cnn.com, apnews.com, economist.com, foxnews.com, latimes.com, axios.com, businessinsider.com, techcrunch.com, wikipedia.org, github.com, quora.com, etsy.com, walmart.com, shopify.com, wix.com, squarespace.com and wordpress.com. One pattern stands out. Six of the 20 sites that block meta-externalagent also name facebookexternalhit in order to allow it, so they decline training but keep their link previews. Only reddit.com blocks link previews, through a * group that disallows everything.

Does meta-externalagent ignore robots.txt?

Meta's doc tells you to block meta-externalagent with robots.txt, uses it as its example, and names only meta-externalfetcher and facebookexternalhit as crawlers that may bypass robots.txt. I have not tested Meta's compliance, and I have no server logs to show. But when a site owner reports that meta-externalagent ignored a block, one of four things is usually worth checking first.

• The block is less than a day old. Meta says its crawlers may cache robots.txt for up to 24 hours.
• The visits are from a different crawler. meta-externalfetcher and facebookexternalhit can bypass robots.txt by Meta's own account, so check the exact user agent in the log.
• The rule is in the wrong group. If meta-externalagent has its own group, it ignores your * group entirely, so a Disallow line placed only under * does not apply to it.
• The visitor is not Meta. Anyone can send the meta-externalagent user agent. The next section shows how to check the address.

If none of those explains it, Meta's doc gives a contact for crawler questions: webmasters@meta.com. Send the date your rule went live, the exact robots.txt group, and a few log lines with time, address and user agent.

How can you tell if a visit is really from Meta?

Check that the visiting address belongs to Meta's network. Meta's crawler doc publishes no IP list for its AI crawlers (checked Oct 31, 2026). Its only mention of IP addresses is in the facebookexternalhit requirements, which call allowlisting by IP "more secure" without listing any. Meta's peering page does name its networks: AS32934, "the global AS," and AS63293.

The routing registry RADb lists the address blocks registered to those networks. This command lists AS32934's:

whois -h whois.radb.net -- '-i origin AS32934'

On Oct 31, 2026, that query returned 1,091 unique address blocks: 437 IPv4 and 654 IPv6, including 31.13.64.0/18 and 2a03:2880::/32. Run the same query for AS63293. An address outside every block in both lists is not in any block registered to the networks Meta's peering page names, whatever its user agent says. Registry data can lag, and Meta registers other networks too, so this is a strong hint, not a verdict. An address inside them shows only that the network is Meta's, not which crawler sent the request. Meta does not document this as a crawler check, so treat it as a sanity check.

What if meta-externalagent is crawling your site too hard?

If the crawler is putting load on your server, block it in robots.txt first and wait a day. Meta's crawler doc does not mention Crawl-delay, and RFC 9309 does not define it, so do not count on a slow-down line working (checked Oct 31, 2026). If the load continues past a day, block or rate-limit the user agent at your host or CDN firewall. A firewall rule works whatever robots.txt says, though it also stops a crawler you might want back later.

Keep facebookexternalhit out of any firewall rule unless you mean it. Blocking it stops link previews on Facebook, Instagram and Messenger. Meta's doc also asks that the crawler be able to fetch a page "within a few seconds," so a slow site can lose its previews even when nothing is blocked. For the bigger question of what to do about AI crawlers in general, including whether to use llms.txt alongside robots.txt, the comparison of llms.txt vs robots.txt explains what each file can and cannot do.