Anthropic
AI training
ClaudeBot
Collects public web content that Anthropic may use to train and improve Claude.
Follows robots.txt- User agent token
- ClaudeBot
- Operator
- Anthropic
- Verification
- Published IP list
- Source
- Anthropic documentation
What an AI training crawler does
A training crawler gathers public pages that may end up in the data used to train AI models. It sends no visitors back, so blocking it is the usual choice for sites that don't want their content used that way.
Anthropic also honors the Crawl-delay extension.
Block or allow ClaudeBot in robots.txt
Add one of these to the robots.txt file at the root of your domain. Crawlers read it before they fetch anything else.
Block ClaudeBot
User-agent: ClaudeBot
Disallow: /Allow ClaudeBot
User-agent: ClaudeBot
Allow: /More from Anthropic
Questions, answered.
ClaudeBot is an AI training crawler run by Anthropic. Collects public web content that Anthropic may use to train and improve Claude.
Yes. Anthropic says ClaudeBot follows robots.txt. Anthropic also honors the Crawl-delay extension.
Add "User-agent: ClaudeBot" and "Disallow: /" to your robots.txt.
Check the request's IP address against the list Anthropic publishes. A user agent alone proves nothing, since any script can copy it.
No. ClaudeBot collects pages for model training and sends no visitors back.
Block it if you don't want your pages used to train models. It sends no visitors, so blocking it costs you no traffic.
In its own documentation at https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler. Every fact on this page comes from there.
Still have questions? We're happy to help.
See which bots really read your site.
NoirTrack counts ClaudeBot and every other bot apart from real visitors, and its firewall can block them before they reach your pages.
Start free trialFree 14-day trial, no credit card. The server SDK also sees bots that never run JavaScript.