Every bot that reads your site, and what it wants.

An AI crawler is a bot an AI company runs to read web pages. Some collect pages to train models, some build the index an AI assistant searches, and some fetch a page because a person asked about it. Each kind wants something different, so each needs its own rule.

Block the training crawlers if you don't want your content used to train models; they send no visitors back. Keep AI search crawlers and assistant fetchers allowed if you want to be cited in AI answers, since those visits come from real people asking about your topic.

No. OpenAI uses GPTBot to collect training data and OAI-SearchBot for ChatGPT search, and each reads its own robots.txt rules. You can block GPTBot and still allow OAI-SearchBot.

It works for the crawlers that follow it, which is most of the ones listed here. Some assistant fetchers may skip it because a person started the request, and a bot that fakes its user agent ignores it. For those, block the request at your server or firewall.

An AI training crawler, like GPTBot or ClaudeBot, collects pages that may train future models and sends no visitors. An AI search crawler, like OAI-SearchBot or PerplexityBot, builds the index an assistant searches before it answers, which is how your pages get cited and linked.

If you want to be found, allow the AI search crawlers and the assistant fetchers, and decide on training crawlers separately. Allowing OAI-SearchBot, Claude-SearchBot and PerplexityBot keeps you in the answers of ChatGPT, Claude and Perplexity.

List each AI crawler in robots.txt under its own User-agent line with "Disallow: /", or build the file with our free robots.txt generator. Then block the fetchers that may skip robots.txt at your firewall.

Check the IP, not the user agent. Most companies publish their crawlers' IP ranges or a reverse DNS name, and each crawler page here says which one applies. The free bot verifier runs the check for you.

Most AI crawlers never run JavaScript, so an analytics script in the page never sees them. They show up in server logs, or with a check that runs on the server before the page loads, such as the NoirTrack server SDK.