Googlebot

Crawls pages for Google Search, Discover, Images, Video and News.

Follows robots.txt

User agent token: Googlebot
Operator: Google
Verification: Published IP list , reverse DNS googlebot.com, google.com
Source: Google documentation

What a search engine crawler does

A search engine crawler indexes pages for regular search results. Blocking it removes the site from that engine, so most sites keep it allowed.

Block or allow Googlebot in robots.txt

Add one of these to the robots.txt file at the root of your domain. Crawlers read it before they fetch anything else.

Block Googlebot

User-agent: Googlebot
Disallow: /

Allow Googlebot

User-agent: Googlebot
Allow: /

More from Google

Google-ExtendedA robots.txt switch that decides whether Google may use your pages to train Gemini models and to ground their answers. GoogleOtherA general crawler Google product teams use to fetch public content outside Search. Google-CloudVertexBotCrawls a site when its owner asks Vertex AI to build an agent from it. Storebot-GoogleCrawls product pages for Google Shopping. Google-InspectionToolFetches a page when someone runs the Rich Results Test or URL Inspection in Search Console. AdsBot-GoogleChecks the quality of landing pages used in Google Ads. Googlebot-ImageCrawls images for Google Images, Discover and the image features in Search. Googlebot-VideoCrawls videos for the video features in Google Search. Googlebot-NewsA robots.txt name for Google News: it crawls with the regular Googlebot user agents, and these rules decide what appears in News. GoogleOther-ImageThe image version of GoogleOther, fetching public image URLs for Google product teams. GoogleOther-VideoThe video version of GoogleOther, fetching public video URLs for Google product teams. AdsBot-Google-MobileChecks the quality of mobile landing pages used in Google Ads. Mediapartners-GoogleReads pages that show AdSense or Ad Manager ads so the ads match the content. APIs-GoogleDelivers push notification messages sent through Google APIs. Google-SafetyCrawls for abuse checks, such as finding malware. Google-AgentBrowses the web and takes actions on pages when a person asks a Google AI agent to. Google-GeminiNotebookFetches a page that a person added as a source in a Gemini Notebook project. FeedFetcher-GoogleFetches RSS and Atom feeds for Google News and WebSub. GoogleProducerProcesses the feeds publishers supply to Google News through Publisher Center. Google-Read-AloudFetches a page so Google can read it out loud with text to speech. GoogleMessagesBuilds the link preview when someone sends your page in Google Messages. Google-PinpointFetches a page that a Pinpoint user added as a source to their research collection. Google-Site-VerificationFetches the token that proves you own a site in Search Console. Google-CWSRequests the URLs a developer lists in a Chrome Web Store extension or theme.

Browse all 66 crawlersAI and search bots from 16 companies, with the rules to block or allow each one.

Questions, answered.

Googlebot is a search engine crawler run by Google. Crawls pages for Google Search, Discover, Images, Video and News.

Yes. Google says Googlebot follows robots.txt.

Add "User-agent: Googlebot" and "Disallow: /" to your robots.txt.

Check the request's IP address against the list Google publishes. A user agent alone proves nothing, since any script can copy it.

Yes. Googlebot indexes pages for Google's search results, which send people to your site.

Usually not. Blocking Googlebot removes your pages from its search results.

In its own documentation at https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers. Every fact on this page comes from there.

Still have questions? We're happy to help.

Read the docs Contact support

See which bots really read your site.

NoirTrack counts Googlebot and every other bot apart from real visitors, and its firewall can block them before they reach your pages.

Start free trial

Free 14-day trial, no credit card. The server SDK also sees bots that never run JavaScript.