meta-webindexer

Reads pages to improve the search results in Meta AI.

Follows robots.txt

User agent token: meta-webindexer
Operator: Meta
Verification: None published
Source: Meta documentation

What an AI search crawler does

An AI search crawler builds the index an assistant searches before it answers. Being in that index is how a page gets quoted and linked in AI answers, which is a growing source of visitors.

Block or allow meta-webindexer in robots.txt

Add one of these to the robots.txt file at the root of your domain. Crawlers read it before they fetch anything else.

Block meta-webindexer

User-agent: meta-webindexer
Disallow: /

Allow meta-webindexer

User-agent: meta-webindexer
Allow: /

More from Meta

meta-externalagentCrawls pages to train Meta's AI models and to improve its products. meta-externalfetcherFetches a link when a person asks Meta AI to open it. facebookexternalhitBuilds the link preview when someone shares your page in one of Meta's apps. meta-externaladsCrawls pages to improve Meta's advertising and business products.

Browse all 66 crawlersAI and search bots from 16 companies, with the rules to block or allow each one.

Questions, answered.

meta-webindexer is an AI search crawler run by Meta. Reads pages to improve the search results in Meta AI.

Yes. Meta says meta-webindexer follows robots.txt.

Add "User-agent: meta-webindexer" and "Disallow: /" to your robots.txt.

Meta publishes no IP list for meta-webindexer, so it can only be recognised by its user agent, which other bots can copy.

It can. Pages meta-webindexer indexes may be cited and linked in Meta's AI answers, and those links bring readers.

Only if you don't want to appear in Meta's AI answers. Blocking meta-webindexer removes your pages from what it can cite.

In its own documentation at https://developers.facebook.com/docs/sharing/webmasters/web-crawlers. Every fact on this page comes from there.

Still have questions? We're happy to help.

Read the docs Contact support

See which bots really read your site.

NoirTrack counts meta-webindexer and every other bot apart from real visitors, and its firewall can block them before they reach your pages.

Start free trial

Free 14-day trial, no credit card. The server SDK also sees bots that never run JavaScript.