Free tools
Free tools for the AI web.
Check what AI crawlers can read, fix your SEO tags, catch fake bots, and work out what your traffic is worth.
AI crawlers and bots 4
See which AI and search crawlers your robots.txt lets in, and which blocks cost you citations.
Open toolPick which AI crawlers may read your site and get a robots.txt to paste in.
Open toolPaste a user agent and IP from your logs to check whether a request really came from the crawler it names.
Open toolBuild an llms.txt for your site, or check the one you have against the format AI tools expect.
Open toolSEO and site checks 5
See your public IP, location, network and browser, or look up any IP address.
Open toolCheck a page's title, description and Open Graph tags, and preview it on Google, X, LinkedIn and chat apps.
Open toolFind the analytics, session replay and ad tracking tags on any website.
Open toolGrade a site's HTTP security headers, from HSTS to Content-Security-Policy, and see what to add.
Open toolMake an IndexNow key and send new or changed URLs to Bing, Yandex and other search engines in one go.
Open toolMarketing and calculators 5
Build campaign links with utm_source, utm_medium and utm_campaign so you know which post or ad sent each visitor.
Open toolWork out your conversion rate from visitors and conversions, and what a better rate is worth.
Open toolWork out return on ad spend, profit after ads, and the break-even ROAS for your margin.
Open toolWork out revenue per visitor, average order value and conversion rate from three numbers.
Open toolWork out bounce rate and engagement rate from sessions and single-page visits.
Open toolKnow every bot by name.
66 AI and search crawlers from 16 companies: who runs each one, whether it follows robots.txt, and the rules to block or allow it.
OpenAI
GPTBot
AI training
Follows robots.txt
OpenAI
OAI-SearchBot
AI search
Follows robots.txt
Anthropic
ClaudeBot
AI training
Follows robots.txt
Google
Googlebot
Search engines
Follows robots.txt
Google
Google-Extended
AI training
robots.txt switch, never crawls
Perplexity
PerplexityBot
AI search
Follows robots.txt
Apple
Applebot
AI search
Follows robots.txt
+59See all crawlers
Questions, answered.
Yes. Every tool works without an account or a card. The generators and calculators run in your browser with no limit; the checks that run on our servers allow 15 a day without an account, and an account removes the daily limit.
Block the AI training crawlers if you don't want your pages used to train models; they send no visitors back. Keep AI search crawlers and assistant fetchers allowed if you want to be cited in AI answers.
Training crawlers like GPTBot and ClaudeBot collect pages that may train future models. AI search crawlers like OAI-SearchBot and PerplexityBot build the index an assistant searches before it answers, which is how your pages get cited.
It works for crawlers that follow it, which includes the big AI and search companies. Some assistant fetchers may skip it, and bad bots ignore it, so a firewall is the only way to stop those.
Check its IP, not its name. Paste the user agent and IP into the bot verifier, which compares the IP with the list the company publishes or its reverse DNS.
llms.txt is a Markdown file that lists a site's most useful pages for AI tools. It is a proposal, not a standard, so it is optional; robots.txt is still what crawlers obey.
Most AI crawlers never run JavaScript, so an analytics script in the page never sees them. They show up in server logs, or with a check that runs on the server, such as the NoirTrack server SDK.
Each check fetches only what it needs from that site, like its robots.txt or one page, and keeps the result for up to an hour so a repeat check is instant. The IPs you look up aren't stored, and nothing you check is collected into a list.
Still have questions? We're happy to help.
See who actually visits your site.
A free tool answers one question once. NoirTrack answers it every day: people, AI crawlers and bots counted apart, and every sale tied to the visit that brought it.
Start free trialFree 14-day trial, no credit card