Article · AI search · Googlebot · CDN · crawling · bot protection

Moving your site to a CDN? Don't let bot protection block Googlebot

On 11 November 2026 Google's Search Central blog warned that moving behind a CDN can briefly push more crawler traffic onto your own server, and that reacting with blanket bot protection can push your pages and shopping listings out of Search. What Google said, and what to check before and after you switch.

The question
Can moving my site to a CDN hurt how Google crawls it?
The short answer
It can, for a short time. On 11 November 2026 Google's Search Central blog explained that a content delivery network (CDN) makes your site respond faster, which tells Google's crawlers they can crawl faster. But a new CDN starts with an empty cache, and crawlers ask for far more of your pages than people do, so those first requests still go to your own server. The bigger risk is the reaction: if blanket bot protection or CAPTCHA challenges are turned on without letting verified crawlers through, Google says its crawlers get stuck on error pages, back off, and your pages and shopping listings will begin to drop out of Search. Our read: before any CDN or firewall change, make sure verified crawlers are allowed, warm the cache for your most important pages, and use a short 429 or 503 response only if your server really needs a break.

TL;DR: On 11 November, Google's Search Central blog described a problem site owners sometimes run into when they move behind a content delivery network (CDN): crawler traffic to their own server can spike right after the switch instead of dropping. A faster site tells Google's crawlers they can crawl faster, and a new CDN's cache is empty, so those requests still reach your server. Google's warning is about the reaction. Blanket bot protection that challenges every automated visitor also stops Google's crawlers, and Google says that, left in place, your pages and shopping listings "will begin to drop out of Search." Our read: a CDN move is a good change, but the firewall settings that go with it deserve the same care as the move itself.

The likely default: the CDN goes live, the server graphs jump, someone sees a wall of bot traffic, and they turn on the strictest bot setting in the dashboard.

That fixes the graph and can quietly cost you search traffic. Google's advice is narrower: let verified crawlers through, block the specific bots you don't want, and slow Google down with a temporary status code if you have to.

What Google said

A CDN serves copies of your pages from servers close to your visitors. Your own server, which Google calls the origin, still holds the original.

Google's crawlers watch how fast and how reliably a site responds. Slow responses lower how hard they crawl. Fast, stable responses raise it, if there is demand to crawl more. Google calls this the crawl capacity limit, also known as hostload. When your DNS points at a CDN, response times drop, and in Google's words, "it looks like your server just got a massive hardware upgrade," so crawlers start asking for more.

The catch is the cache. A brand-new CDN has no copies yet. The first request for any URL at a given edge location, whether a page, a script or an image, is a cache miss, and the CDN has to fetch it from your origin. People tend to visit the same popular pages, which get cached quickly. Crawlers do the opposite: their job is to work through the breadth of your site. Until the cache fills up, Google writes, "your CDN isn't acting as a shield yet." Google also notes that crawlers don't ramp up immediately and many CDNs pace connections, which softens the spike.

Then the part that matters most for a small or mid-size business. When a CDN or web application firewall (WAF) meets automated traffic with a challenge such as a CAPTCHA, Google's crawlers fail it. With no exemption for verified crawlers, they are "stuck on an error page." To Google, the site looks "deliberately restricted or broken." Google says its crawlers will back off and your pages and shopping listings "will begin to drop out of Search." The post also notes that those error pages aren't useful for Search or "other products relying on crawled data," without naming them.

Google's three recommendations:

  1. If you block bots, allow verified ones. Most CDNs and firewalls have a setting or managed rule to allow known or verified bots, and Google's crawlers can be identified through its published IP ranges. Better still, Google says, use robots.txt or rules aimed at the specific bots you don't want, instead of blocking all automated visitors.
  2. Warm the cache before switching all traffic. Request your highest-traffic pages, upcoming promotional or seasonal pages, and your most-crawled URLs through the CDN at a controlled pace first.
  3. Use a 429 or 503 status as a short brake. If crawling still overloads your server while the cache warms up, return 429 (Too Many Requests) or 503 (Service Unavailable) "for a short window (a few hours, or 1-2 days at most)."

What the crawl-rate guide adds

The blog post points to Google's guide to reducing the crawl rate. Read it before you use that brake, because it is broader than it sounds.

  • Returning 500, 503 or 429 on a significant number of URLs slows crawling across the whole hostname, including pages that are still returning content. Crawling picks up again once the errors drop.
  • With a 503 or 429 you can add a Retry-After header that says when to come back.
  • Google warns that slowing crawling means Googlebot discovers fewer new pages, existing pages are refreshed less often (so prices and product availability can take longer to update in Search), and removed pages may stay indexed longer. For Google Ads, campaigns "may be canceled or paused, and your ads may not serve."
  • Don't keep it up longer than 1 to 2 days. If Googlebot sees these codes on the same URL for multiple days, the URL "may be dropped from Google's index."

The guide also says sudden crawling spikes often come from the site itself: faceted navigation and filters, calendars that generate a URL for every date, or Dynamic Search Ad targets, and it recommends checking your server's recent access logs to see whether one of those is the source. Our read: check those logs before assuming the CDN is the only cause.

What it means for a business

This section is our read, not Google's.

Most small and mid-size sites never think about crawl budget. Google's own crawl budget guide says that if your pages are crawled the same day they are published, you don't need to worry about it. That's why this post is useful: the risk isn't how much Google crawls. It is a security setting that tells Google your site is broken.

CDN moves are often handled by whoever runs hosting or IT, and no one looks at search until later, if at all. The bot setting is usually one switch in a dashboard. People can usually get past a challenge, so the site looks fine to the team while crawlers are stuck.

The same applies beyond a migration. Any time someone tightens bot rules, for example after a scraping scare or to keep AI bots out, ask whether verified crawlers are still allowed. If you want to keep some AI companies from training on your content, a targeted robots.txt rule is the tool for that. We covered how that works at Google, Bing, OpenAI and Anthropic. A blanket block is a different thing. AI agents acting for customers raise the same question, as in our piece on ChatGPT agents that can pay on a site.

What to do this week

  1. Find out who controls your CDN and firewall settings. If it's a vendor or an agency, ask them whether verified bots are allowed and whether any rule serves CAPTCHA or challenge pages to automated visitors.
  2. Turn on the verified-bot exemption if your provider has one, before any stricter bot setting. Block unwanted bots by name, not all of them.
  3. Check that a "Googlebot" in your logs is really Google. Google's verification guide describes a reverse and forward DNS lookup, or matching the IP address against Google's published lists. Anyone can put "Googlebot" in a user agent.
  4. Look at the Crawl Stats report in Search Console after any change. The report shows crawl requests, response codes, average response time and host status. Our advice: check that host status, which includes whether Google could fetch your robots.txt file, stays healthy, and look in the crawl responses table for a jump in client errors (4XX) or server errors (5XX) right after a firewall change. Google says sites with fewer than a thousand pages usually don't need it, but it is the quickest place to see this problem.
  5. Plan a CDN move like a launch. Warm the cache for your top pages, seasonal pages and most-crawled URLs first, and watch your server load for the first few days.
  6. Treat 429 or 503 as a short brake, not a setting. A few hours, 1 to 2 days at most, and remember it slows crawling across the whole hostname.

Sources

Want to know what AI says about your business?

We check it for you.