Anthropic
AI search
Claude-SearchBot
Crawls pages to improve the search results Claude cites.
Follows robots.txt- User agent token
- Claude-SearchBot
- Operator
- Anthropic
- Verification
- Published IP list
- Source
- Anthropic documentation
What an AI search crawler does
An AI search crawler builds the index an assistant searches before it answers. Being in that index is how a page gets quoted and linked in AI answers, which is a growing source of visitors.
Block or allow Claude-SearchBot in robots.txt
Add one of these to the robots.txt file at the root of your domain. Crawlers read it before they fetch anything else.
Block Claude-SearchBot
User-agent: Claude-SearchBot
Disallow: /Allow Claude-SearchBot
User-agent: Claude-SearchBot
Allow: /More from Anthropic
Questions, answered.
Claude-SearchBot is an AI search crawler run by Anthropic. Crawls pages to improve the search results Claude cites.
Yes. Anthropic says Claude-SearchBot follows robots.txt.
Add "User-agent: Claude-SearchBot" and "Disallow: /" to your robots.txt.
Check the request's IP address against the list Anthropic publishes. A user agent alone proves nothing, since any script can copy it.
It can. Pages Claude-SearchBot indexes may be cited and linked in Anthropic's AI answers, and those links bring readers.
Only if you don't want to appear in Anthropic's AI answers. Blocking Claude-SearchBot removes your pages from what it can cite.
In its own documentation at https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler. Every fact on this page comes from there.
Still have questions? We're happy to help.
See which bots really read your site.
NoirTrack counts Claude-SearchBot and every other bot apart from real visitors, and its firewall can block them before they reach your pages.
Start free trialFree 14-day trial, no credit card. The server SDK also sees bots that never run JavaScript.