AdsBot-Google

GooglebotCrawls pages for Google Search, Discover, Images, Video and News. Google-ExtendedA robots.txt switch that decides whether Google may use your pages to train Gemini models and to ground their answers. GoogleOtherA general crawler Google product teams use to fetch public content outside Search. Google-CloudVertexBotCrawls a site when its owner asks Vertex AI to build an agent from it. Storebot-GoogleCrawls product pages for Google Shopping. Google-InspectionToolFetches a page when someone runs the Rich Results Test or URL Inspection in Search Console. Googlebot-ImageCrawls images for Google Images, Discover and the image features in Search. Googlebot-VideoCrawls videos for the video features in Google Search. Googlebot-NewsA robots.txt name for Google News: it crawls with the regular Googlebot user agents, and these rules decide what appears in News. GoogleOther-ImageThe image version of GoogleOther, fetching public image URLs for Google product teams. GoogleOther-VideoThe video version of GoogleOther, fetching public video URLs for Google product teams. AdsBot-Google-MobileChecks the quality of mobile landing pages used in Google Ads. Mediapartners-GoogleReads pages that show AdSense or Ad Manager ads so the ads match the content. APIs-GoogleDelivers push notification messages sent through Google APIs. Google-SafetyCrawls for abuse checks, such as finding malware. Google-AgentBrowses the web and takes actions on pages when a person asks a Google AI agent to. Google-GeminiNotebookFetches a page that a person added as a source in a Gemini Notebook project. FeedFetcher-GoogleFetches RSS and Atom feeds for Google News and WebSub. GoogleProducerProcesses the feeds publishers supply to Google News through Publisher Center. Google-Read-AloudFetches a page so Google can read it out loud with text to speech. GoogleMessagesBuilds the link preview when someone sends your page in Google Messages. Google-PinpointFetches a page that a Pinpoint user added as a source to their research collection. Google-Site-VerificationFetches the token that proves you own a site in Search Console. Google-CWSRequests the URLs a developer lists in a Chrome Web Store extension or theme.