Google’s Updated Crawler Guidance Recommends ETags via @sejournal, @martinibuster

Google announced an update to their crawler documentation, adding more information about caching which should help better understand how to optimize for Google’s crawler. By following the new guidelines on implementing proper HTTP caching headers, SEOs and publishers can improve crawling efficiency and optimize server resources.

Updated Crawler Documentation

The crawler documentation now has a section that explains how Google’s crawlers use HTTP caching mechanisms that help to conserve computing resources for both publishers and Google during crawling.

Additions to the documentation significantly expand on the prior version.

Caching Mechanisms

Google recommends enabling caching with headers like ETag and If-None-Match, as well as optionally Last-Modified and If-Modified-Since, to signal whether content has changed. This can help reduce unnecessary crawling and save server resources, which is a win for both publishers and Google’s crawlers.

The new documentation states:

“Google’s crawling infrastructure supports heuristic HTTP caching as defined by the HTTP caching standard, specifically through the ETag response- and If-None-Match request header, and the Last-Modified response- and If-Modified-Since request header.”

Google’s Preference For Preference for ETag

Google recommends using ETag over Last-Modified because ETag is less prone to errors like date formatting issues and provides more precise content validation. It also explains what happens if both ETag and Last-Modified response headers are served:

“If both ETag and Last-Modified response header fields are present in the HTTP response, Google’s crawlers use the ETag value as required by the HTTP standard.”

The new documentation also states that other HTTP caching directives are not supported.

Variable Support Across Crawlers

The new documentation explains that support for caching differs among Google’s crawlers. For example, Googlebot supports caching for re-crawling, while Storebot-Google has limited caching support.

Google explains:

“Individual Google crawlers and fetchers may or may not make use of caching, depending on the needs of the product they’re associated with. For example, Googlebot supports caching when re-crawling URLs for Google Search, and Storebot-Google only supports caching in certain conditions”

Guidance On Implementation

Google’s new documentation recommends contacting hosting or CMS providers for assistance. It also suggests (but doesn’t require) that publishers set the max-age field of the Cache-Control response header in order to help crawlers know when to crawl specific URLs.

Entirely New Blog Post

Google has also published a brand new blog post:

Crawling December: HTTP caching

Read the updated documentation:

HTTP Caching

Featured Image by Shutterstock/Asier Romero

Google CEO: Search Will Change Profoundly In 2025 via @sejournal, @martinibuster

Sundar Pichai, Google’s CEO, was interviewed by Andrew Ross Sorkin at the New York Times DealBook Summit, where he discussed what to expect from Google Search in 2025 but also struggled to articulate Google’s concern for content creators.

When asked to compare where Google is today relative to the rest of the industry and whether Google should be the “default winner” Pichai reminded the interviewer that these were “the earliest stages of a profound shift” and underlined that Google is a leader in AI and not the follower. The entire AI industry is built on top of Google research discoveries that were subsequently open sourced, particularly transformers, without which the AI industry would not exist as it is today.

Pichai answered:

“Look, it’s a such a dynamic moment in the industry. When I look at what’s coming ahead, we are in the earliest stages of a profound shift. We have taken such a deep full stack approach to AI.

…we do world class research. We are the most cited, when you look at gen AI, the most cited… institution in the world, foundational research, we build AI infrastructure and when I’m saying AI infrastructure all the way from silicon, we are in our sixth generation of tensor processing units. You mentioned our product reach, we have 15 products at half a billion users, we are building foundational models, and we use it internally, we provide it to over three million developers and it’s a deep full stack investment.

We are getting ready for our next generation of models, I just think there’s so much innovation ahead, we are committed to being at the state of the art in this field and I think we are. Just coming today, we announced groundbreaking research on a text and image prompt creating a 3D scene. And so the frontier is moving pretty fast, so looking forward to 2025.”

Blue Link Economy And AI

It was pointed out by the interviewer that Google was the first mover on AI and then it wasn’t (a reference to OpenAI’s breakout in 2022 and subsequent runaway success). He asked Pichai how much of that was Google protecting the “blue link economy” so as not “to hurt or cannibalize that business” which is worth hundreds of billions of dollars.

Pichai answered that out of all the projects at Google, AI was applied the most to Search, citing BERT, MUM and multimodal search as helping close the gaps in search quality. Something that some in the search industry fail to understand is that AI has has been a part of Google since 2012 when it used Deep Neural Networks for identifying images and speech recognition and in 2014 when it introduced the world to sequence to sequence learning (PDF) for understanding strings of text. In 2015 Google introduced RankBrain, an AI system directly related to ranking search results.

Pichai  answered:

“The area where we applied AI the most aggressively, if anything in the company was in search, the gaps in search quality was all based on Transformers internally. We call it BERT and MUM and you know, we made search multimodal, the search quality improvements, we were improving the language understanding of search. That’s why we built Transformers in the company.

So and if you look at the last couple of years, we have with AI overviews, Gemini is being used by over a billion users in search alone.”

Search Will Change Profoundly In 2025

Pichai continued his answer, stating directly that Search will profoundly change not just in 2025, but in early 2025. He also said that progress is going to get harder because the easier things to innovate have been done (low hanging fruit).

He said:

“And I just feel like we are getting started. Search itself will continue to change profoundly in 2025. I think we are going to be able to tackle more complex questions than ever before. You know, I think we’ll be surprised even early in 2025, the kind of newer things search can do compared to where it is today… “

Pichai also said that progress wouldn’t be easy:

“I think the progress is going to get harder when I look at 2025, the low hanging fruit is gone.

But I think where the breakthroughs need to come from where the differentiation needs to come from is is your ability to achieve technical breakthroughs, algorithmic breakthroughs, how do you make the systems work, you know, from a planning standpoint or from a reasoning standpoint, how do you make these systems better? Those are the technical breakthroughs ahead.”

Is Search Going Away?

The interviewer asked Pichai if Google has leaned into AI enough, quoting an author who suggested that Google’s “core business is under siege” because people are increasingly getting answers from AI and other platforms outside of search, and that the value of search would be “deteriorating” because so much of the content online will be AI-generated.

He answered that it’s precisely in a scenario where the Internet is flooded with inauthentic content that search becomes even more valuable.

Pichai answered:

“In a world in which you’re flooded with like lot of content …if anything, something like search becomes more valuable. In a world in which you’re inundated with content, you’re trying to find trustworthy content, content that makes sense to you in a way reliably you can use it, I think it becomes more valuable.

To your previous part about there’s a lot of information out there, people are getting it in many different ways. Look, information is the essence of humanity. We’ve been on a curve on information… when Facebook came around, people had an entirely new way of getting information, YouTube, Facebook, Tik… I can keep going on and on.

…I think the problem with a lot of those constructs is they are zero sum in their inherent outlook. They just feel like people are consuming information in a certain limited way and people are all dividing that up. But that’s not the reality of what people are doing. “

Pichai Stumbles On Question About Impact On Creators

The interviewer next asked if content is being devalued. He used the example of someone who researches a topic for a book, reads twenty books, cites those sources in the bibliography and then gets it published. Whereas Google ingests everything and then “spits” out content all day long, defeating the human who in earlier times would write a book.

Andrew Ross Sorkin said:

“You get to spit it out a million times. A million times a day. And I just wonder what the economics of that should be for the folks that create it in the beginning.”

Sundar Pichai defended Google by saying that Google spends a lot of time thinking about the impact to the “ecosystem” of publishers and how much traffic it sends to them. The interviewer listened to Sundar’s answer without mentioning the elephant in the room, search results stuffed with Reddit and advertising that crowds out content created by actual experts, and the de-prioritization of news content which has negatively impacted traffic to news organizations around the world.

It was at this point that Pichai appeared to stumble as he tried to find the words to respond. He avoids mentioning websites, speaking in the abstract about the “ecosystem” and then when he runs out of things to say changes course and begins speaking about how Google compensates copyright holders who sign up for YouTube’s Content ID program.

He answered:

“Look I… uh… It’s a… very important question… uhm… look I… I… think… I think more than any other company… look you know… we for a long time through… you know… be it in search making sure… while it’s often debated, we spend a lot of time thinking about the traffic we send to the ecosystem.

Even through the moment through the transition over the past couple of years. It’s an important priority for us.”

At this point he started talking about Google’s content platform YouTube and how they use “Content ID” which is used to identify copyright-protected content. Content ID is a program that benefits the corporate music, film, and television industries, copyright owners who “own exclusive rights to a substantial body of original material that is frequently uploaded to YouTube.”

Pichai continued:

“In YouTube we put a lot of effort into understanding and you know identifying content and with content ID and uh creating monetization for creators.

I think… I think those are important principles, right. I think um… there’s always going to be a balance between understanding what is fair use uh… when new technology comes versus how do you… give value back proportionate to the value of the IP, the hard work people have put in.”

Insightful Interview Of Alphabet’s CEO

The interviewer did a great job at asking the hard questions but I think many in the search marketing community who are more familiar with the search results would have asked follow up questions about content creators who are not on Google’s YouTube platform or the non-expert content that pushes down content by actual experts.

Watch the New York Times Interview here:

Featured Image by Shutterstock/Shutterstock AI Generator
(no irony intended)