Google Explains How CDNs Impact Crawling & SEO via @sejournal, @martinibuster

Google published an explainer that discusses how Content Delivery Networks (CDNs) influence search crawling and improve SEO but also how they can sometimes cause problems.

What Is A CDN?

A Content Delivery Network (CDN) is a service that caches a web page and displays it from a data center that’s closest to the browser requesting that web page. Caching a web page means that the CDN creates a copy of a web page and stores it. This speeds up web page delivery because now it’s served from a server that’s closer to the site visitor, requiring less “hops” across the Internet from the origin server to the destination (the site visitor’s browser).

CDNs Unlock More Crawling

One of the benefits of using a CDN is that Google automatically increases the crawl rate when it detects that web pages are being served from a CDN. This makes using a CDN attractive to SEOs and publishers who are concerned about increasing the amount of pages that are crawled by Googlebot.

Normally Googlebot will reduce the amount of crawling from a server if it detects that it’s reaching a certain threshold that’s causing the server to slow down. Googlebot slows the amount of crawling, which is called throttling. That threshold for “throttling” is higher when a CDN is detected, resulting in more pages crawled.

Something to understand about serving pages from a CDN is that the first time pages are served they must be served directly from your server. Google uses an example of a site with over a million web pages:

“However, on the first access of a URL the CDN’s cache is “cold”, meaning that since no one has requested that URL yet, its contents weren’t cached by the CDN yet, so your origin server will still need serve that URL at least once to “warm up” the CDN’s cache. This is very similar to how HTTP caching works, too.

In short, even if your webshop is backed by a CDN, your server will need to serve those 1,000,007 URLs at least once. Only after that initial serve can your CDN help you with its caches. That’s a significant burden on your “crawl budget” and the crawl rate will likely be high for a few days; keep that in mind if you’re planning to launch many URLs at once.”

When Using CDNs Backfire For Crawling

Google advises that there are times when a CDN may put Googlebot on a blacklist and subsequently block crawling. This effect is described as two kinds of blocks:

1. Hard blocks

2. Soft blocks

Hard blocks happen when a CDN responds that there’s a server error. A bad server error response can be a 500 (internal server error) which signals a major problem is happening with the server. Another bad server error response is the 502 (bad gateway). Both of these server error responses will trigger Googlebot to slow down the crawl rate. Indexed URLs are saved internally at Google but continued 500/502 responses can cause Google to eventually drop the URLs from the search index.

The preferred response is a 503 (service unavailable), which indicates a temporary error.

Another hard block to watch out for are what Google calls “random errors” which is when a server sends a 200 response code, which means that the response was good (even though it’s serving an error page with that 200 response). Google will interpret those error pages as duplicates and drop them from the search index. This is a big problem because it can take time to recover from this kind of error.

A soft block can happen if the CDN shows one of those “Are you human?” pop-ups (bot interstitials) to Googlebot. Bot interstitials should send a 503 server response so that Google knows that this is a temporary issue.

Google’s new documentation explains:

“…when the interstitial shows up, that’s all they see, not your awesome site. In case of these bot-verification interstitials, we strongly recommend sending a clear signal in the form of a 503 HTTP status code to automated clients like crawlers that the content is temporarily unavailable. This will ensure that the content is not removed from Google’s index automatically.”

Debug Issues With URL Inspection Tool And WAF Controls

Google recommends using the URL Inspection Tool in the Search Console to see how the CDN is serving your web pages. If the CDN firewall, called a Web Application Firewall (WAF), is blocking Googlebot by IP address you should be able to check for the blocked IP addresses and compare them to Google’s official list of IPs to see if one of them are on the list.

Google offers the following CDN-level debugging advice:

“If you need your site to show up in search engines, we strongly recommend checking whether the crawlers you care about can access your site. Remember that the IPs may end up on a blocklist automatically, without you knowing, so checking in on the blocklists every now and then is a good idea for your site’s success in search and beyond. If the blocklist is very long (not unlike this blog post), try to look for just the first few segments of the IP ranges, for example, instead of looking for 192.168.0.101 you can just look for 192.168.”

Read Google’s documentation for more information:

Crawling December: CDNs and crawling

Featured Image by Shutterstock/JHVEPhoto

Google Says Temporal Anomalies Affect Googlebot Crawl via @sejournal, @martinibuster

Google’s John Mueller answered a question on Reddit about Googlebot crawling and the completeness of the snapshot. The person asking the question received a response that touched on edge cases and temporal anomalies in crawling.

A Googlebot “screenshot” refers to a representation of what a web page looks like to Googlebot.

What a web page looks like depends on how it renders the page after executing JavaScript, loading CSS and downloading necessary images.

Google Search Console’s URL inspection tool gives an idea of what a web page looks like to Google. This tool helps publishers and SEOs understand how Google “sees” a web page.

Question About Knowing What Googlebot “Sees”

The person asking the question was talking about Googlebot screenshots. What they apparently meant was the rendered page as Googlebot itself sees it.

This is the question the Redditor asked:

“Is the Googlebot screenshot a complete picture of what Google can see?”

They later clarified with the following answers to questions:

“How can I know what google see in my article? …I want to know what Googlebot see in my website.”

Is Googlebot Screenshot A Complete Picture?

Returning to the original question of whether the “Googlebot screenshot a complete picture of what Google can see,” Google’s John Mueller offered the following answer.

“For the most part, yes. But there are some edge cases and temporal anomalies. Tell us more about what you’re trying to check.”

Mueller’s response acknowledges that the Googlebot screenshot represents what Google sees when it crawls a page.

Temporal Anomalies In Googlebot Screenshot

The person asking the question referred to a Googlebot screenshot as what Googlebot “sees” when it visits a web page. That also seems to be the context of Mueller’s answer.

Mueller’s answer referred to temporal anomalies which could be a reference to temporary issues at the time the web page was crawled that could have effected what resources were downloaded and consequently affected how the web page looked to Googlebot in that moment.

Google Search Console’s URL Inspection Tool also provides a snapshot that shows a live preview of how a web page appears to Google. It’s a good way to check if everything is rendered by Google the way it’s supposed to look.

Read the discussion on Reddit:

Is the Googlebot screenshot a complete picture of what Google can see?

Featured Image by Shutterstock/Sammby

Google’s December 2024 Updates: Final Results via @sejournal, @martinibuster

Google launched two updates in December that rolled out in succession like waves carrying happy surfers to the beach and a tsunami to the unlucky few. Many SEOs who braced for SERP drama were met with a surprising impact.

Facebook Groups: The Update Was Good!

A common theme in Facebook and forum posts is that the Google Update had a positive effect that resulted in many sites hit by previous sites were returning to life all by themselves. One person reported that a number of their dormant affiliate sites suddenly awakened and were attracting traffic. Then there were some random SEOs taking credit for de-ranked client sites returning to the SERPs, leading me to wonder if they also take credit for their clients losing rankings in the first place…

Black Hat Response To Google’s Update

The forums on Black Hat World (BHW) generally provide an indicator of how much hurt Google’s updates are handing out. Some members of BHW forum were posting about how their sites were coming back.

One member posted:

“This new update is the first update in past years for my websites to grow 🙂 For now, traffic has increased 100%+ and keeps going up…”

Another one echoed that post:

“For me it is going good for 2 days. Hope not to be bad…”

A third member shared:

“My website ranking returned today after 9 months from March update. So seams the keywords and links showing again for first time since. I was affected by the march and thought it wouldn’t recover ever again hopefully this December update once finished rolling out my site won’t be removed again”

And yet another one shared a similar experience:

“My dead website is starting to pick up keywords on ahrefs, increasing impressions on search console, but still no major change regarding traffic.”

There was one outlier who shared that their financial site lost rankings likely due to having been built on expired domains (though it’s easy to imagine other reasons).

Among the celebratory sharing was this outlier post from someone whose glass is half empty:

“Lots of sites got tanked including 3 of mine.”

Overall there was a positive tone to the black hat forum members sharing their actual experiences with Google’s December updates. That fits the pattern of what was being shared in Facebook groups, that the December updates were bringing some sites back from whatever algorithm hits they suffered in previous updates.

Then There Is The X Response.

On X, every Google update announcement is met by a large amount of comments about how big brands are the winners, spam sites are dominating the SERPs (which contradicts the first complaint), and that Google is destroying small businesses. These kinds of commments have been around for decades, from before X/Twitter. The defining characteristic of these kinds of remarks are that they are general in nature and don’t reflect anything specific about the update, they’re just venting about Google.

The response to Google’s update announcement on X was predictably negative. I’m not being snarky when I say the responses are predictably negative, that’s literally the consistent tone for every update announcement that Google makes on X.

Representative posts:

Google is destroying small businesses:

“…How is it possible every update since September 23 has destroyed the traffic of small/medium publishers while boosting corp websites that put out top 10s trash content.”

Google updates consistently benefit Reddit:

“Just call it what it is: The Gift More Traffic to Reddit Update #94”

And the Google destroyed my business post:

“I used to work independently, writing blogs and earning a decent income. I even believed I would never need a traditional job in my life. But thanks to Google, which wiped out my websites one after another, I’m now left with no income, no motivation and no job. Thank you Google”

Again, I am not minimizing the experiences of the people making those posts. I am just pointing out that none of those posts reflect experiences specific to the December Google updates. They are general statements about Google’s algorithms.

There are some outliers who were posting that the update was big but the same people say that for every update. Although 2024 was a year of massive change, those outlier posts have been consistent from well before 2024. I don’t know what’s going on there.

Google’s Core Algorithm Update: What Happened?

It feels clear that Google dialed back something in the algorithm that was suppressing the rankings of many websites. It’s been my opinion that Google’s algorithms that determine if a site is “made for search engines” has been overly hard on expert sites by people whose poor understanding of SEO resulted in otherwise high quality sites laden with high traffic keywords, sometimes showing exact matches to keywords shown in People Also Asked. That, in my opinion results in a “made for search engines” kind of look. Could it be that Google tweaked that algorithm to be a little more forgiving of content spam just as it’s forgiving for link spam?

Something to consider is that this update was followed by an anti-spam update, which could be an improved classifier to catch the spam sites that may have been set loose by the core algorithm update, while leaving the expert sites in the search results.

What About 2025?

Google’s CEO recently stated that 2025 would be a year of major changes. If the two December updates are representative of what’s coming in the future it could be that the heralded changes may not be as harsh as the series of updates in 2024. We can hope, right?

Google Reveals Top Searches Of 2024 via @sejournal, @MattGSouthern

Google has released its annual “Year in Search” report, highlighting the top trending searches, news, people, entertainment, and more that captured the world’s attention in 2024.

Based on Google search data, the report offers a glimpse into the topics, events, and figures that defined the year.

Here’s an overview of top trending searches worldwide and a breakdown of U.S.-specific trends.

Global Top Trending Searches 2024

Overall Searches

Globally, the top trending searches were dominated by major sporting events.

The Copa América soccer tournament topped the list, followed by the UEFA European Championship and the ICC Men’s T20 World Cup cricket tournament.

1. Copa América.
2. UEFA European Championship.
3. ICC Men’s T20 World Cup.
4. India vs. England.
5. Liam Payne.

News

In the news category, the U.S. presidential election was the most searched topic worldwide, surpassing interest in the Olympic games, which came in third.

Excessive heat and Hurricane Milton also drove significant search interest, ranking as the second and fourth top global news events.

1. U.S. Election.
2. Excessive heat.
3. Olympics.
4. Hurricane Milton.

Entertainment

In entertainment, veteran comedian and actor Katt Williams topped the global list of most searched actors. Rising stars like Ella Purnell and Hina Khan also earned spots in the top five.

Highly anticipated sequels dominated movie searches, with “Inside Out 2,” “Deadpool & Wolverine,” and “Dune: Part Two” ranking as the top three.

Top Actors:

1. Katt Williams.
2. Pawan Kalyan.
3. Adam Brody.
4. Ella Purnell.
5. Hina Khan.

Top Movies:

1. Inside Out 2.
2. Deadpool & Wolverine.
3. Saltburn.
4. Beetlejuice Beetlejuice.
5. Dune: Part Two.

United States Top Trending Searches 2024

Overall Searches

1. Election.
2. Donald Trump.
3. Connections.
4. New York Yankees.
5. Kamala Harris.

People

1. Donald Trump.
2. Kamala Harris.
3. JD Vance.
4. Joe Biden.
5. Catherine, Princess of Wales.

Entertainment

Actors:

1. Katt Williams.
2. Jacob Elordi.
3. Glen Powell.
4. Jeremy Allen White.
5. Shane Gillis.

Musicians

1. Usher.
2. Diddy.
3. Kendrick Lamar.
4. Drake.
5. Justin Timberlake.

Sports

Sports fans worldwide were captivated by boxer Mike Tyson, the second most searched athlete globally behind Algerian karate champion Imane Khelif.

Sixteen-year-old soccer phenom Lamine Yamal also made the top five, reflecting the growing buzz about the young star.

Among sports teams, American franchises dominated, with the New York Yankees, Los Angeles Dodgers, and Boston Celtics ranking in the top five for global searches.

However, interest in soccer was evident, with Inter Miami CF and German club Bayer Leverkusen also making the list.

Athletes:

1. Mike Tyson.
2. Imane Khalif.
3. Simone Biles.
4. Scottie Scheffler.
5. Jake Paul.

Teams:

1. New York Yankees.
2. Los Angeles Dodgers.
3. New York Mets.
4. Boston Celtics.
5. Minnesota Timberwolves.

Digital Culture & Gaming

In gaming, the massively multiplayer life sim Connections was the most searched game globally, followed by the creature-capturing open-world adventure Palworld.

The trends point to gamers’ desire for connection and imaginative escapism.

Games:

1. Connections.
2. Strands.
3. Infinite Craft.
4. Palworld.
5. Sprunki.

Lifestyle Trends

Aesthetics:

1. Mob wife aesthetic.
2. Nancy Meyers aesthetic.
3. Pillow face aesthetic.
4. Brat aesthetic.
5. Decora aesthetic.

Food & Recipes

Olympic-themed chocolate muffin recipes topped global food-related searches, while the traditional Chinese treat tanghulu came in second. Dubai chocolate bar rounded out the top five, showcasing an international palate.

1. Olympic chocolate muffins.
2. Tanghulu.
3. Tini’s mac and cheese.
4. Mama Kelce’s cookie.
5. Dense bean salad.

Popular Locations (Google Maps)

When it came to getting out and about, Google Maps searches showed Central Park in New York as the most searched-for park in the world, while the British Museum topped the list of museums.

For sports venue searches, Madison Square Garden led the pack.

Top U.S. Hiking Areas:

1. Dater Mountain Nature Park, Sloatsburg, NY.
2. Columbia River Gorge National Scenic Area, OR.
3. Garden of the Gods, Colorado Springs, CO.
4. Looking Glass Falls, Asheville, NC.
5. Diamond Head Crater Trailhead, Honolulu, HI.

Music Trends

Google’s “Hum to Search” feature, which identifies songs based on audio clips, revealed that people around the world were eager to locate catchy tunes like “Beautiful Things” by Benson Boone and “Die With A Smile” by Lady Gaga.

The sea shanty “Wellerman” also had a global resurgence.

Most Hummed Songs:

1. Beautiful Things – Benson Boone.
2. Axel F – Crazy Frog
3. Wellerman – Santiano.
4. Past Lives – Slushii.
5. Too Sweet – Hozier.

A Snapshot Of 2024 Through Search

Google’s Year in Search 2024 reflects what connected people worldwide over the past year.

Key highlights include the World Cup, travel, a renewed love for nature, and virtual adventures. Common themes are sports, entertainment, connection, and comfort.

We’ll be watching to see what people search for in 2025 and beyond.

For a complete overview of this year’s search trends, visit Google’s Year in Search mini-site.

More Resources:

Featured Image: Daniel Pawer/Shutterstock