How to Whitelist Googlebot: Verify and Allow Legitimate Crawlers

What you need before you start

Whitelisting means you only allow traffic from IPs you’ve confirmed belong to Googlebot. To do this safely, you need:

• Server or firewall access – where you can add allow rules (e.g., .htaccess, nginx, Apache, AWS WAF, Cloudflare).
• Google’s published IP ranges – Google lists them in its Googlebot IP ranges document. Keep this list current because ranges can change.
• Basic command-line access – you’ll run dig or nslookup to check reverse DNS.
• A test URL – a sample page you can fetch with a browser or tool to confirm Googlebot still visits.

If you’re using a CDN or WAF, many have built-in “known bot” toggles. Those can handle verification automatically, but you still need to confirm the bot is really Googlebot before trusting any toggle.

Verify the IP with reverse DNS

The first step is to check that the IP address making the request actually belongs to Google. Reverse DNS maps an IP back to a hostname. Googlebot’s hostnames always end in googlebot.com or google.com.

Run a reverse DNS lookup on the suspicious IP. On most systems, use:

dig -x 66.249.65.200

Replace the example IP with the one from your logs. If the result looks like crawl-66-249-65-200.googlebot.com, it’s a strong signal the visitor is Googlebot. If the hostname is something else – or doesn’t resolve at all – do not whitelist it.

This step catches the most common fake bots: scripts that claim a Googlebot user agent but come from a random IP. Reverse DNS gives you objective evidence the IP is under Google’s control.

Confirm the hostname against Google’s public list

Reverse DNS alone isn’t enough. An attacker who controls a DNS record can make an IP resolve to a googlebot.com domain. So the next step is to check the returned hostname against Google’s official verification list.

Fetch Google’s current list of Googlebot IP ranges and hostnames. Google provides this at https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot. Use a tool like curl or a web browser to access it. Look for the IP you’re verifying and confirm the exact hostname matches.

Additionally, Googlebot’s user-agent string includes a reference like Googlebot/2.1, but user agents can be forged. Never trust the user agent alone. The DNS check is the authoritative test.

If the IP resolves to a hostname like crawl-66-249-65-200.googlebot.com and that IP appears in Google’s published ranges, you can safely whitelist it. If either check fails, treat the traffic as suspicious.

Add verified IPs to your whitelist

Once you’ve confirmed a set of Googlebot IPs, add them to your server, firewall, or CDN allowlist. The exact method depends on your stack. Here are common examples.

Apache or Nginx

For Apache, add an Allow rule inside a <Directory> block or an <IfModule> directive. For Nginx, use the allow directive in the server block. You’ll typically combine this with a deny all rule for everyone else – but only if you want to block all other traffic. Most people whitelist only specific paths, like search engine landing pages, rather than the whole site.

Cloudflare or other CDN

Many CDNs have a “bot management” section where you can toggle “Known Bots.” Enabling this automatically verifies Googlebot via DNS on each request. You can also create a firewall rule that allows only verified Googlebot IPs. Check your CDN’s documentation for the exact steps.

AWS WAF or similar

In AWS WAF, create a rule that checks the IP address against a set you define. You can import Google’s published ranges as a managed rule or maintain a custom list. The key is to use the same verification logic – reverse DNS plus hostname match – before adding an IP.

Whitelisting is most useful when you’re facing heavy bot traffic that looks like Googlebot. For example, if you see many requests with a Googlebot user agent but from IPs that fail the DNS check, you can block those and allow only the verified ones.

Test that Googlebot can still crawl your site

After implementing the whitelist, verify that real Googlebot traffic still reaches your site. The easiest way is to check Google Search Console’s URL Inspection tool. Enter a sample URL and request a fetch. Google will use its real crawler, and you can see for the “Fetch succeeds” message.

Alternatively, use your server logs. Look for hits from the IPs you whitelisted. If they appear with a 200 status code, your rule is working. If you see 403 or 404 errors for those IPs, your whitelist is too strict or misconfigured.

Also check that robots.txt is not blocking Googlebot inadvertently. A whitelist at the network level works together with robots.txt, not as a replacement. Both must allow Googlebot.

Common mistakes when whitelisting Googlebot

Most whitelisting failures come from skipping DNS verification. Here are the mistakes to avoid.

• Trusting the user agent – Any bot can set its user agent to “Googlebot.” Never use it as a whitelist condition.
• Using stale IP lists – Google changes its IP ranges. Subscribe to Google’s update feed or re-check monthly.
• Whitelisting a whole /8 or /16 – Googlebot IPs are spread across many ranges. Whitelisting a broad block can let malicious traffic through.
• Forgetting about other legitimate bots – Bingbot, Yandex, and others need separate verification. A whitelist for Googlebot doesn’t cover them.
• Ignoring CDN caching – If you use a CDN, the request IP might be the CDN’s edge, not the original bot. You must verify the source IP or enable the CDN’s bot verification feature.

When whitelisting alone isn’t enough

Whitelisting only manages who gets in. It doesn’t tell you whether the traffic is legitimate. A sophisticated bot can sometimes pass the DNS check if it controls a compromised IP or DNS record. That’s why security teams combine whitelisting with behavioral checks.

BotRefund’s detection approach is a good example. It runs 106 independent checks that look at browser behavior, network patterns, device signals, and session activity. One anomaly is not a verdict; the system cross-references everything. This corroboration catches bots that look like real users – even if their IP passes a simple DNS check.

So use whitelisting as a first gate, but not as your only defense. If you operate an ad-heavy site or a lead form, add behavior-based filtering to stop fake visitors that slip through.

Key facts about bot verification

FAQ

Does whitelisting Googlebot affect my rankings?

No. Googlebot will still crawl and index your site if the whitelist is correctly configured. In fact, whitelisting the real Googlebot can improve crawl efficiency because you remove noise.

How often do Googlebot IP ranges change?

Google updates its ranges occasionally. Check Google’s official documentation at least monthly, and subscribe to any change notifications from Google.

Can I whitelist all Google-owned bots at once?

Google owns many bots (Googlebot for search, AdSense crawler, etc.). Google provides a combined list, but each bot has its own verification method. Usually the same DNS check works for all google.com/googlebot.com hostnames, but verify per bot.

What about other search engines like Bing?

They require separate DNS verification. Bingbot hostnames end in search.msn.com. Repeat the same process for each legitimate bot you need to allow.

Is whitelisting necessary if I use a WAF like Cloudflare?

Cloudflare’s “Known Bots” feature automates DNS verification. You still may want a custom rule for fine-grained control, but the built-in toggle is a good start.

What if I accidentally block Googlebot?

Google Search Console will show crawl errors. You can fix the rule and re-fetch the URL to recover. In most cases, rankings recover once Googlebot can crawl again.