OpenAI
AI search
OAI-SearchBot
Indexes pages so they can appear and be linked in ChatGPT search answers.
Follows robots.txt- User agent token
- OAI-SearchBot
- Operator
- OpenAI
- Verification
- Published IP list
- Source
- OpenAI documentation
What an AI search crawler does
An AI search crawler builds the index an assistant searches before it answers. Being in that index is how a page gets quoted and linked in AI answers, which is a growing source of visitors.
Block or allow OAI-SearchBot in robots.txt
Add one of these to the robots.txt file at the root of your domain. Crawlers read it before they fetch anything else.
Block OAI-SearchBot
User-agent: OAI-SearchBot
Disallow: /Allow OAI-SearchBot
User-agent: OAI-SearchBot
Allow: /More from OpenAI
Questions, answered.
OAI-SearchBot is an AI search crawler run by OpenAI. Indexes pages so they can appear and be linked in ChatGPT search answers.
Yes. OpenAI says OAI-SearchBot follows robots.txt.
Add "User-agent: OAI-SearchBot" and "Disallow: /" to your robots.txt.
Check the request's IP address against the list OpenAI publishes. A user agent alone proves nothing, since any script can copy it.
It can. Pages OAI-SearchBot indexes may be cited and linked in OpenAI's AI answers, and those links bring readers.
Only if you don't want to appear in OpenAI's AI answers. Blocking OAI-SearchBot removes your pages from what it can cite.
In its own documentation at https://developers.openai.com/api/docs/bots. Every fact on this page comes from there.
Still have questions? We're happy to help.
See which bots really read your site.
NoirTrack counts OAI-SearchBot and every other bot apart from real visitors, and its firewall can block them before they reach your pages.
Start free trialFree 14-day trial, no credit card. The server SDK also sees bots that never run JavaScript.