Meta

AI assistants

meta-externalfetcher

Fetches a link when a person asks Meta AI to open it.

May skip robots.txt
User agent token
meta-externalfetcher
Operator
Meta
Verification
None published
Source
Meta documentation

What an AI assistant fetcher does

An assistant fetcher visits only when someone asks an AI assistant a question that needs your page. Each visit stands for a real person, often one who is about to click through, so blocking it usually costs you readers.

Meta says it may bypass robots.txt, because a person asked for the page.

Block or allow meta-externalfetcher in robots.txt

Add one of these to the robots.txt file at the root of your domain. Crawlers read it before they fetch anything else.

Block meta-externalfetcher

User-agent: meta-externalfetcher
Disallow: /

Allow meta-externalfetcher

User-agent: meta-externalfetcher
Allow: /

More from Meta

meta-externalagentCrawls pages to train Meta's AI models and to improve its products. meta-webindexerReads pages to improve the search results in Meta AI. facebookexternalhitBuilds the link preview when someone shares your page in one of Meta's apps. meta-externaladsCrawls pages to improve Meta's advertising and business products.
Browse all 66 crawlersAI and search bots from 16 companies, with the rules to block or allow each one.

Questions, answered.

meta-externalfetcher is an AI assistant fetcher run by Meta. Fetches a link when a person asks Meta AI to open it.

Not always. meta-externalfetcher fetches pages for a person, and Meta says robots.txt may not apply. Meta says it may bypass robots.txt, because a person asked for the page.

Add "User-agent: meta-externalfetcher" and "Disallow: /" to your robots.txt. Since robots.txt may not stop it, block it at your firewall or server too.

Meta publishes no IP list for meta-externalfetcher, so it can only be recognised by its user agent, which other bots can copy.

Indirectly. meta-externalfetcher opens your page because a person asked about it, and the answer can link back to you.

Usually not. Each visit stands for a person asking about your page, so blocking meta-externalfetcher turns readers away.

In its own documentation at https://developers.facebook.com/docs/sharing/webmasters/web-crawlers. Every fact on this page comes from there.

Still have questions? We're happy to help.

Read the docs Contact support

See which bots really read your site.

NoirTrack counts meta-externalfetcher and every other bot apart from real visitors, and its firewall can block them before they reach your pages.

Start free trial

Free 14-day trial, no credit card. The server SDK also sees bots that never run JavaScript.