Googlebot User Agent: The Full List, How to Verify It, and How to Control It

Any bot can claim to be Googlebot. Typing that string into a user agent header takes seconds, and most servers never check any further, so a traffic spike or an odd crawl pattern labeled "Googlebot" in your logs might not be Google at all.

This guide cross-checks Google's own crawler documentation against the top-ranking pages on this topic, since most of them list outdated user agent strings and skip verification entirely.

You'll get the current list of every Google crawler user agent, the exact commands to confirm a request is genuinely Google, a diagnostic flow for suspicious traffic, and how to control or safely throttle each crawler without hurting your indexing. Here's where to start.

What Is a Googlebot User Agent (And Why the String Alone Doesn't Prove Anything)

A Googlebot user agent is the text string that Google's crawler sends in the HTTP request header to identify itself to your server, something like Mozilla/5.0 ... (compatible; Googlebot/2.1; +http://www.google.com/bot.html). Every browser and bot sends one of these strings, and it's how a server decides how to treat the request.

Here's the part that trips people up: the user agent string is just self-reported text. Nothing stops a scraper, a script, or a competitor's bot from setting its own user agent to say "Googlebot." The string on its own is not proof of anything.

Google itself splits its automated clients into three categories:

• Common crawlers like Googlebot, which respect robots.txt and do the automatic crawling that feeds Search.
• Special-case crawlers, like AdsBot, that operate under a specific agreement between Google and the site owner and don't always follow the same rules as Googlebot.
• User-triggered fetchers, activated when someone uses a Google tool like the URL Inspection tool or Rich Results Test against a specific page.

Googlebot is one client among several, and treating any string containing "Googlebot" as automatically trustworthy is how spoofed traffic gets waved through. The rest of this guide covers the current list of strings, how to actually verify one, and what to do once you know what you're looking at.

The Complete List of Google Crawler User Agents

Most lists of Googlebot user agents online are missing the crawlers Google has added in the last few years, particularly the ones tied to AI training data and Vertex AI. Here's the current set, current as of this writing (verify against Google's own crawler list if you need the latest, since Google updates it periodically):

Why the W.X.Y.Z shows up in the Googlebot strings: since Google made Googlebot "evergreen" in 2019, its rendering engine tracks the live version of Chrome, and the user agent string updates its version number to match. If you're comparing a string from an old blog post against what you see in your own logs today, a different Chrome version number doesn't mean anything is wrong. Match on the structure and the Googlebot/2.1 token, not the exact Chrome version.

How to Verify a Request Is Actually Googlebot (Not Spoofed)

Checking the user agent string tells you what a request claims to be. It doesn't tell you what it is. The only reliable way to confirm a request genuinely came from Google is to check where it actually came from, not what it says about itself.

In practice: the standard method is a reverse DNS lookup on the source IP, followed by a forward DNS lookup to confirm the result. Here's the sequence, using a real IP from Google's own documented Googlebot range as the example:

1. Run a reverse lookup on the IP from your logs.

host 66.249.66.1

You should get back a hostname. If it's genuinely Googlebot, that hostname will end in .googlebot.com or .google.com. For this example, it resolves to crawl-66-249-66-1.googlebot.com.

2. Run a forward lookup on that hostname to confirm it matches.

host crawl-66-249-66-1.googlebot.com

This should return the same IP you started with. If it doesn't match, or if the reverse lookup in step 1 didn't return a .googlebot.com/.google.com hostname at all, the request isn't from Google, no matter what its user agent string says.

Checking many IPs at once: if you're auditing a log file with hundreds of requests claiming to be Googlebot, doing this one IP at a time doesn't scale. Google publishes the full list of Googlebot's IP ranges as a JSON file at developers.google.com/search/apis/ipranges/googlebot.json, which you can pull into a script and match your log IPs against directly, no DNS lookups required.

Diagnostic Flow: Suspicious Traffic Claiming to Be Googlebot

Verification only matters if you know when to run it. Here's a symptom-to-fix flow for the situations that actually come up:

How to Control What Each Googlebot Variant Can Access

Every crawler in the table above has its own robots.txt token, and blocking the wrong one is a common, avoidable mistake. Blocking Googlebot broadly also blocks Googlebot-Image and Googlebot-Video, since they inherit rules from the general Googlebot group unless you write a more specific rule for them.

To block a single crawler variant without touching the others, address its token directly:

User-agent: Googlebot-Image
Disallow: /

This removes your images from Google Images while leaving your pages fully crawlable for web search.

The nuance most guides get wrong: Google-Extended. Blocking Google-Extended in robots.txt stops Google from using your content to train its generative AI models (the systems behind Gemini). It does not remove your pages from Google Search, and it does not opt you out of appearing in AI Overviews, since those pull from your existing Search index rather than from Google-Extended's training access. Site owners commonly block this one thinking it also affects their organic visibility. It doesn't.

User-agent: Google-Extended
Disallow: /

If your issue is the opposite problem, pages getting blocked when you didn't intend it, that's usually a robots.txt misconfiguration rather than anything crawler-specific. The indexed though blocked by robots.txt and blocked due to access forbidden (403) guides cover that side of it.

How to Slow Down an Aggressive Googlebot Without Hurting Your Rankings

If genuine, verified Googlebot traffic is straining your server, the instinct is to block it. Don't. Blocking stops your pages from being indexed. What you actually want is a slower crawl rate, not zero crawl.

The short-term lever: returning HTTP 500, 503, or 429 status codes tells Googlebot to back off. But this only works for a limited window, roughly two days, before Google resumes its normal crawl rate regardless of whether you're still returning errors. It's a way to buy yourself time, not a fix you can leave in place.

The durable fix: use Search Console's crawl rate and crawl stats reporting to flag the issue to Google directly, rather than relying on server errors as your only lever. If you leave 5xx or 429 responses in place past that short window, you risk a different problem entirely: pages dropping out of the index because Google can't successfully fetch them. The server error (5xx) guide covers what that looks like once it happens.

Practical Ways to Put the Googlebot User Agent to Work

Beyond one-time verification, the Googlebot user agent is useful for two ongoing habits.

Viewing your site the way Googlebot sees it. In Chrome DevTools, open the three-dot menu, go to More tools > Network conditions, uncheck "Use browser default," and select a Googlebot user agent from the dropdown (or paste one from the table above). Reload the page. This is the fastest way to catch rendering differences between what a real visitor sees and what gets served to the crawler.

Filtering server logs to see where Googlebot actually spends its time. Once you've verified which log entries are genuinely Googlebot (using the DNS check above, not the user agent string alone), filter your access logs down to just those requests and group by URL path:

grep "Googlebot" access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

That gives you the URLs Googlebot hit most often over the log window, which is the real, concrete version of "crawl budget," not an abstract concept. Cross-reference that list against Search Console's Crawl Stats report to catch discrepancies between what your logs show Googlebot fetching and what's actually making it into the index. That comparison is exactly the kind of gap SEO Heatmap is built to surface visually from your Search Console data, instead of exporting and cross-checking two reports by hand.

If you're already pulling Search Console data to make sense of crawl and indexing patterns, SEO Heatmap turns that data into a visual map so the gaps stand out without manual cross-checking. Give it a try, it's free.

Frequently asked questions

How do I verify that a request claiming to be Googlebot is real?

Run a reverse DNS lookup on the IP address that made the request, then a forward DNS lookup on the hostname it returns. If the hostname ends in .googlebot.com or .google.com and the forward lookup resolves back to the original IP, the request is genuine. If either check fails, treat the traffic as unverified no matter what the user agent string says.

Is it normal to see several different Googlebot user agents in my server logs?

Yes. Google runs separate crawlers for web search, images, video, news, and ads, and each one sends its own user agent string. Seeing Googlebot-Image alongside the standard Googlebot desktop and smartphone strings in the same log file is expected, not a sign of a problem.

What's the difference between Googlebot and Google-Extended?

Googlebot crawls and indexes your pages for Google Search. Google-Extended is a separate robots.txt token that controls whether your already-crawled content can be used to train Google's generative AI models, and it doesn't run its own crawl. Blocking Google-Extended stops AI training use without removing your pages from Search results.

Can blocking Googlebot's user agent stop bad bots that copy it?

No. Blocking traffic based on the user agent string alone won't stop a bot that's deliberately faking that string, since it can just as easily ignore your rule. Verify the source IP with a reverse and forward DNS lookup first, then block by IP address or firewall rule if it fails to check out.

Will slowing down Googlebot with 5xx errors hurt my rankings?

It can, if you leave the errors in place too long. Returning 500, 503, or 429 responses reduces Googlebot's crawl rate for roughly two days, but Google resumes its normal rate after that regardless. If the 5xx responses continue past that window, pages Google can't successfully fetch risk dropping out of the index.

How can I see my site the way Googlebot sees it?

Open Chrome DevTools, go to More tools, then Network conditions, and switch the user agent to one of Googlebot's strings. Reload the page to compare what a real visitor sees against what gets served to the crawler. This is a fast way to catch rendering or content differences before they affect indexing.

What should I do if I find fake Googlebot traffic on my site?

Confirm it's fake first with a reverse and forward DNS lookup on the source IP, since a spoofed user agent alone isn't proof either way. Once confirmed, block the traffic by IP address or firewall rule rather than by user agent string, and keep monitoring your logs for the same pattern from new IPs.

Does blocking Googlebot-Image also block the regular Googlebot crawler?

No, and the reverse mistake is more common: blocking the general Googlebot token in robots.txt also blocks Googlebot-Image and Googlebot-Video unless you write separate rules for them. To block only images, target the Googlebot-Image token directly and leave the general Googlebot rule untouched.

Verifying traffic and reading crawl patterns one log line at a time gets old fast. SEO Heatmap turns your Search Console data into a visual map, so you can spot indexing and crawl gaps across your whole site without digging through raw reports. It's free to try.