Google Confirms: High-Quality Content Is Crawled More Often via @sejournal, @MattGSouthern

SEO professionals have long discussed the concept of a “crawl budget,” which refers to the limited number of pages search engines can crawl daily.

The assumption is that sites must stay within this allotted budget to get pages indexed. In a recent podcast, Google search engineers debunked some misconceptions about crawl budget and shed light on how Google prioritizes crawling.

How Googlebot Prioritizes Crawling

“I think there’s a lot of myths out there about crawling, about what it is and what it isn’t. And things like crawl budgets and phrases you hear thrown around that may be quite confusing to people,” said Dave Smart, an SEO consultant and Google Product Expert, during the podcast.

So, how does Google decide what to crawl?

“You need to do it by looking at what’s known, finding somewhere to start, a starting point. And from that, you get the links and stuff, and then you would try and determine what’s important to go and fetch now, and maybe what can wait until later and maybe what’s not important at all,” explained Smart.

Gary Illyes from Google’s search relations team agreed with this framework.

“If search demand goes down, that also correlates to the crawl limit going down. So if you want to increase how much we crawl, you somehow have to convince search that your stuff is worth fetching,” he said.

The key, then, is to produce content that Google recognizes as valuable based on user interaction.

Focus On Quality & User Experience

“Scheduling is very dynamic. As soon as we get the signals back from search indexing that the quality of the content has increased across this many URLs, we would just start turning up demand,” said Illyes.

This means there is no fixed “budget” that sites must adhere to. Improving page quality and proving usefulness to searchers can overcome any assumed limitations.

No One-Size-Fits-All Approach

“We don’t have an answer for every site,” Illyes admitted regarding crawl prioritization. “If you improved that section, then probably it’s going to help a lot.”

According to Google, the bottom line is to Focus on producing high-quality content rather than trying to reverse engineer a non-existent crawl quota. Earning links naturally and better serving users will take care of the rest.

Hear the full discussion in the podcast episode linked below:

Featured Image: BestForBest/Shutterstock

The Saga Of John Mueller’s Freaky Robots.txt via @sejournal, @martinibuster

The robots.txt file of the personal blog of Google’s John Mueller became a focus of interest when someone on Reddit claimed that Mueller’s blog had been hit by the Helpful Content system and subsequently deindexed. The truth turned out to be less dramatic than that but it was still a little weird.

SEO Subreddit Post

The saga of John Mueller’s robots.txt started when a Redditor posted that John Mueller’s website was deindexed, posting that it fell afoul of Google’s algorithm. But as ironic as that would be that was never going to be the case because all it took was a few seconds to get a load of the website’s robots.txt to see that something strange was going on.

Here’s the top part of Mueller’s robots.txt which features a commented Easter egg for those taking a peek.

The first bit that’s not seen every day is a disallow on the robots.txt. Who uses their robots.txt to tell Google to not crawl their robots.txt?

Now we know.

The next part of the robots.txt blocks all search engines from crawling the website and the robots.txt.

So that probably explains why the site is deindexed in Google. But it doesn’t explain why it’s still indexed by Bing.

I asked around and Adam Humphreys, a web developer and SEO(LinkedIn profile), suggested that it might be that Bingbot hasn’t been around Mueller’s site because it’s a largely inactive website.

Adam messaged me  his thoughts:

“User-agent: *
Disallow: /topsy/
Disallow: /crets/
Disallow: /hidden/file.html

In those examples the folders and that file in that folder wouldn’t be found.

He is saying to disallow the robots file which Bing ignores but Google listens to.

Bing would ignore improperly implemented robots because many don’t know how to do it. “

Adam also suggested that maybe Bing disregarded the robots.txt file altogether.

He explained it to me this way:

“Yes or it chooses to ignore a directive not to read an instructions file.

Improperly implemented robots directions at Bing are likely ignored. This is the most logical answer for them. It’s a directions file.”

The robots.txt was last updated sometime between July and November of 2023 so it could be that Bingbot hasn’t seen the latest robots.txt. That makes sense because Microsoft’s IndexNow web crawling system prioritizes efficient crawling.

One of directories blocked by Mueller’s robots.txt is /nofollow/ (which is a weird name for a folder).

There’s basically nothing on that page except some site navigation and the word, Redirector.

I tested to see if the robots.txt was indeed blocking that page and it was.

Google’s Rich Results tester failed to crawl the /nofollow/ webpage.

John Mueller’s Explanation

Mueller appeared to be amused that so much attention was being paid to his robots.txt and he published an explanation on LinkedIn of what was going on.

He wrote:

“But, what’s up with the file? And why is your site deindexed?

Someone suggested it might be because of the links to Google+. It’s possible. And back to the robots.txt… it’s fine – I mean, it’s how I want it, and crawlers can deal with it. Or, they should be able to, if they follow RFC9309.”

Next he said that the nofollow on the robots.txt was simply to stop it from being indexed as an HTML file.

He explained:

“”disallow: /robots.txt” – does this make robots spin in circles? Does this deindex your site? No.

My robots.txt file just has a lot of stuff in it, and it’s cleaner if it doesn’t get indexed with its content. This purely blocks the robots.txt file from being crawled for indexing purposes.

I could also use the x-robots-tag HTTP header with noindex, but this way I have it in the robots.txt file too.”

Mueller also said this about the file size:

“The size comes from tests of the various robots.txt testing tools that my team & I have worked on. The RFC says a crawler should parse at least 500 kibibytes (bonus likes to the first person who explains what kind of snack that is). You have to stop somewhere, you could make pages that are infinitely long (and I have, and many people have, some even on purpose). In practice what happens is that the system that checks the robots.txt file (the parser) will make a cut somewhere.”

He also said that he added a disallow on top of that section in the hopes that it gets picked up as a “blanket disallow” but I’m not sure what disallow he’s talking about. His robots.txt file has exactly 22,433 disallows in it.

He wrote:

“I added a “disallow: /” on top of that section, so hopefully that gets picked up as a blanket disallow. It’s possible that the parser will cut off in an awkward place, like a line that has “allow: /cheeseisbest” and it stops right at the “/”, which would put the parser at an impasse (and, trivia! the allow rule will override if you have both “allow: /” and “disallow: /”). This seems very unlikely though.”

And there it is. John Mueller’s weird robots.txt.

Robots.txt viewable here:

https://johnmu.com/robots.txt

Google Search Console Adds INP Metric In Core Web Vitals Report via @sejournal, @MattGSouthern

Google has announced that Interaction to Next Paint (INP), a new metric for measuring website interactivity, is now included as a key element in the Search Console’s Core Web Vitals report.

As of March 12, INP replaced First Input Delay (FID) as a Core Web Vital, signaling a shift in how Google evaluates user experience.

The INP metric, introduced as an experimental measure in May 2022, captures the time between a user’s interaction with a page (such as clicking a button) and when the browser can render the resulting changes on the screen.

This approach aims to provide a more comprehensive assessment of interactivity than FID, which only measured the time to first paint after the initial user interaction.

Evolving Web Metrics For Better User Experience

Google’s Web Vitals initiative, launched in 2018, provides developers with metrics to help optimize critical aspects of user experience. FID was one of the original metrics introduced as part of this effort. However, over time, Google recognized FID’s limitations in fully capturing interactivity, leading to the development of INP.

After a transition period as a ‘pending metric,’ INP replaces FID as a Core Web Vital. This change reflects Google’s ongoing commitment to refining its methods for evaluating and improving web user experience.

Adapting To The INP Transition

With the INP transition approaching, web developers are advised to assess their website’s current INP performance and take steps to optimize for the new metric.

To evaluate current INP scores, you can use tools like PageSpeed Insights and Chrome’s User Experience Report. Google recommends aiming for the “good” threshold, representing performance at the 75th percentile of page loads.

Developers should then diagnose and address issues impacting INP, such as long-running JavaScript tasks, excessive main thread activity, or overly complex DOM structures.

Implications For Web Development & Search Rankings

The adoption of INP as a Core Web Vital has implications for web development practices and SEO.

As Googe incorporates Core Web Vitals into its ranking systems, websites with strong INP scores may see positive changes in search rankings and user engagement metrics.

Web development practices may evolve to prioritize optimizing interaction readiness. This might require developers to re-evaluate application architectures, streamline code, and refine design elements to minimize interaction delays.

In Summary

By replacing the FID metric with INP, Google aims to offer a more comprehensive assessment of website interactivity.

As you navigate this transition, you can now use Search Console to monitor INP performance and take steps to address any issues that may be impacting scores.

Featured Image: BestForBest/Shutterstock

Google Now Says Core Web Vitals Used In Ranking Systems via @sejournal, @martinibuster

Google updated their Page Experience Documentation in order to make it explicitly clear that Core Web Vitals are used by their ranking systems. What’s curious about the change is that Google continues to not say that Core Web Vitals are a ranking factor.

Googlers And Statements About Ranking Factors

Something kind of weird about Googlers is that it seems like they tend not to use the phrase ranking factor. For example, I did a site:search of former Google engineer Matt Cutts’ blog and I couldn’t find a single instance of him writing the phrase “ranking factor” in any of his posts. Same with his YouTube videos when talking about links like here and here.

John Mueller did say that Core Web Vitals was a ranking factor three years ago on Reddit in reference to the Core Web Vitals (CWV), but Google’s Page Experience In Search Results explainer never explicitly says Core Web Vitals are ranking factors.

Which brings us to Google’s SearchLiaison who caused a stir in February 2024 when he tweeted that Google’s documentation didn’t say that Core Web Vitals (CWV) were a ranking factor or a signal.

He tweeted:

“And people go “Well, what does ranking really mean. Maybe it’s signals? They didn’t say it’s not signals!”

So do we have a signal page experience signal? No. That’s why we made a page that says “There is no single signal.”

Oh but wait, so you have multiple signals? Yes, we anticipated this question which is why we have on that same page “Our core ranking systems look at a variety of signals.”

Which leads to things like “So is CWV a signal and if I don’t meet those, am I doomed?” Which is why that same page says “However, great page experience involves more than Core Web Vitals.”

We don’t list what is and isn’t a ranking signal on that page because things change. Maybe something was once; maybe it shifts but aligns with other things we might do to understand page experience. We’re trying to guide people toward some useful resources and things to thing about with page experience but in the end — do whatever you think is providing a great experience for your visitors.”

And in another tweet on the following day he wrote (referring to the Page Experience In Search explainer):

“I didn’t say we have a page experience “ranking signal” nor do we have some single signal like that. The page below specifically says we do NOT have something like that.

“Is there a single “page experience signal” that Google Search uses for ranking?

There is no single signal. Our core ranking systems look at a variety of signals that align with overall page experience.

We don’t say there’s one particular thing people need to do, nor do we say if you don’t do a particular think, you won’t rank. We say look across a range of things and try to provide a good page experience to your visitors”

SearchLiaison is right. The Page Experience In Search Results explainer document didn’t say that Core Web Vitals is a ranking factor, not even in 2022 when it was first published.

Google Almost Says CWV Is A Ranking Factor

After all the explaining without acknowledging Core Web Vitals as a ranking factor and two years of opaqueness in their Page Experience In Search Results documentation about CWV in relation to ranking factors, Google changed their mind and updated their documentation to almost say that Core Web Vitals are a ranking factor.

This is the ambiguous part that was removed from the documentation:

“What aspects of page experience are used in rankings?
There are many aspects to page experience, including some listed on this page. While not all aspects may be directly used to inform ranking, they do generally align with success in search ranking and are worth attention.”

The above passage was replaced with this new paragraph:

“What aspects of page experience are used in ranking?
Core Web Vitals are used by our ranking systems. We recommend site owners achieve good Core Web Vitals for success with Search and to ensure a great user experience generally.

Keep in mind that getting good results in reports like Search Console’s Core Web Vitals report or third-party tools doesn’t guarantee that your pages will rank at the top of Google Search results; there’s more to great page experience than Core Web Vitals scores alone. These scores are meant to help you to improve your site for your users overall, and trying to get a perfect score just for SEO reasons may not be the best use of your time.”

The new documentation doesn’t use the phrase “ranking factor” or “ranking signal” in reference to the core web vitals. But it now explicitly acknowledges that CWV is used by Google’s ranking systems, which is less ambiguous than the previous statement that high CWV scores are recommended for “success with Search.”

Read Google’s updated documentation:

Understanding page experience in Google Search results

Featured Image by Shutterstock/Graphic Grid

Google’s March 2024 Core Update Impact: Hundreds Of Websites Deindexed via @sejournal, @MattGSouthern

The impact of Google’s March 2024 Core Update, which targets low-quality content and spam, is becoming apparent as the rollout progresses.

Recent analyses reveal that over 800 websites have been completely deindexed from Google’s search results in the early stages of the update.

Widespread Deindexing

Ian Nuttall, a prominent figure in the SEO community, has been tracking the indexing status of 49,345 websites following the launch of the March 2024 Core Update and its associated manual actions against AI-generated spam.

Nuttall’s findings shed light on the sweeping impact of Google’s latest initiative.

Of the 49,345 sites monitored, 837 websites had been removed entirely from Google’s search index.

I checked the index status of 49,345 sites to measure the impact of the AI spam manual actions!

1.7% of sites got hit, 20+ million monthly organic visits gone and over $446k in display ad revenue lost.

Want the data? Follow, like, and reply with 👏 and I’ll send it over! pic.twitter.com/sUo22DejaL

— Ian Nuttall (@iannuttall) March 8, 2024

This represents approximately 1.7% of the websites in Nuttall’s database. The deindexed sites span various categories and ad networks, highlighting the broad scope of Google’s actions.

However, it’s too early to tell whether these websites will remain deindexed, as several are already reappearing in Google’s search results.

I ran my script again to check the sites that were deindexed and 21 of them are showing pages indexed in Google again.

The big winners were fresherslive·com and oldtimemusic·com who have 5m and 2.5m pages indexed.

Here’s the full list:

fresherslive·com – 5,050,000 pages… pic.twitter.com/zgpm1toudt

— Ian Nuttall (@iannuttall) March 11, 2024

Significant Traffic & Revenue Losses

The consequences of being deindexed are severe for the affected websites.

According to Nuttall’s analysis, the 837 deindexed sites accounted for over 20.7 million organic search visits per month before the manual action. The sudden loss of traffic is devastating for the impacted websites.

Further, the deindexing is estimated to result in a monthly loss of $446,552 displayed advertising revenue across the affected sites.

Targeting AI-Generated Spam

The findings from Nuttall’s analysis align with a recent study conducted by Originality.ai, which investigated the prevalence of AI-generated content among the deindexed websites.

The study found that 100% of the affected sites showed signs of AI-generated content, with 50% having 90-100% of their posts generated by AI.

These results suggest that Google’s March 2024 Core Update effectively targets websites relying heavily on AI-generated content to manipulate search rankings.

The manual actions taken against these sites demonstrate Google’s commitment to combating the rise of AI-driven spam and low-quality content.

Reshaping The Search Landscape

As the March 2024 Core Update continues to roll out, its impact on the search landscape is becoming increasingly evident.

The widespread deindexing of websites engaging in manipulative practices, particularly those utilizing AI-generated content, signals a significant shift in Google’s approach to maintaining search result quality.

The consistency between Nuttall’s data and the Originality.ai study reinforces that Google is taking decisive action to address the growing problem of AI-driven spam.

As the update progresses, websites and content creators must adapt to Google’s heightened focus on quality and originality.

The full impact of the March 2024 Core Update will become more evident as it continues to roll out over the coming weeks. However, the early indications from Nuttall’s analysis and the Originality.ai study suggest that Google’s efforts to reshape the search landscape and prioritize high-quality, human-generated content are underway.

Google’s Algorithm Hates AI Content? How To Make Google Love It via @sejournal, @martinibuster

Google’s March 2024 core algorithm update is penalizing sites made with AI generated content and it’s a fact that AI-generated content cannot meet the quality standards that Google’s  various documentations outline. But there is still a way to use AI in a way that results in high quality content.

Why AI Cannot Meet Google’s Quality Thresholds

Several ranking systems, including the Reviews and the Helpful Content systems explicitly have quality standards that inherently make it impossible for AI authored content to satisfy.

The addition of an extra E in E-A-T (for experience) should have been a signal to content creators that using AI carried risks.

Examples of SERP Features, Quality Signals and Ranking Signals That Inherently Exclude AI Content

The writing on the wall about AI content has always been in plain sight.

Here are some qualities that Google’s documentations say are important that rule out purely AI-generated content:

• Experience
• Published reviews must be hands-on
• Google News emphasizes human authors in Google News SERPs
• Google Perspectives, announced in May 2023 emphasizes human authors (hidden gems) found in forums
• Author page (expertise questions)
• Author background information (expertise questions)
• Author About page (expertise questions)

Concepts Of Quality

Google published Self-Assessment Questions to help publishers identify if their content measures up to Google’s standards of quality.

These questions don’t list specific ranking factors. They only list concepts of things that in general reflect what high quality websites tend to show.

If AI-generated content cannot fit into those concepts then it’s likely that the content does not meet quality standards, regardless if the publishers try to fake the outward signs of quality like author pages and so on.

Authorship And Expertise

The Expertise section of the Self-Assessment documentation mentions authors in a way that cannot be replicated by machine-generated content.

This section states:

“Does the content present information in a way that makes you want to trust it, such as clear sourcing, evidence of the expertise involved, background about the author or the site that publishes it, such as through links to an author page or a site’s About page?”

The above quoted section centers expertise on the following three factors:

1. Sourcing (citations to sources, fact checking, attribution of quotes)
2. Evidence of expertise involved
3. Author background

Those three qualities are the outward signs typically associated with expertise which is not attainable by AI.

Content Quality: Originality

The content and quality section of the self-assessment guide requires originality.

Here’s what that section of Google’s documentation asks:

“Does the content provide original information, reporting, research, or analysis?
…Does the content provide insightful analysis or interesting information that is beyond the obvious?”

Unoriginality is the hallmark of generative AI. Content created by generative AI is literally the likeliest series of words on any given topic.

First-Hand Expertise

The people-first section of the self-assessment questions asks about first-hand expertise:

“Does your content clearly demonstrate first-hand expertise and a depth of knowledge (for example, expertise that comes from having actually used a product or service, or visiting a place)?”

Clearly a machine doesn’t have first-hand expertise. It can’t handle a product or use a service.

AI Can Still Be Used For Content Creation

Given how many sites with AI-generated content are getting slapped with manual actions during the March 2024 core algorithm update, it may be time to reconsider the place of AI for web content.

There is still a way to use AI that can result in high quality people-first content. What matters most about content is the insight behind the content, not who or what wrote it.

A way forward may be to use a mix of human insight and experience as data that the AI can use for generating content.

How To Create Review Content With AI

For example, it’s possible to scale product reviews by creating a checklist of data points that consumers need in order to make a buying decision. Someone still has to handle the product and review it, but they just have to write scores and comments for each data point on the review checklist.

If the review is of a children’s bicycle then benchmark the things that users want to know about the bicycle like what age and size does the bicycle fit, how much does it weigh, how sturdy are the training wheels and so on. If it’s a television review then the checklist will have benchmarks relating to the richness of the black levels, off-center viewing, ease of setting the colors, and so on.

At the end of the checklist have a section called final impressions that lists pros and cons as well as the overall sentiment where the reviewer writes if they feel positive, neutral, negative about the product and who they feel the product is best for like people on a budget, those who crave performance and so on. Once that’s done upload the document to your AI and ask it to write the review.

How To Write Any Kind Of Content With AI

An acquaintance shared a tip with me about using AI to polish rough content.  His workflow consists of a dictating everything that needs to be said into a recording, without regard for paragraph structure, simply poured out into the recording.  He then uploads it to ChatGPT and asks it to transform it into a professional document. He can even ask it to generate pros and cons and an executive summary.

AI Amplifies Human Input

My suggestion is to think of AI as a ghost writer that takes a rough document and turns it into a polished essay or article. This approach can work for nearly any scenario, including for scaled product descriptions.

The important qualities of content are the ones provided by a human that an AI is incapable of, things like sourcing, evidence of expertise, sourcing and the background that a human brings to a the topic that is being written about. Humans bring the Experience, Expertise, Authoritativeness and Trustworthiness.  AI can take those elements provided by the human and turn it into high quality content.

Given how many sites with AI-generated content are getting slapped with manual actions during the March 2024 core algorithm update, it may be time to reconsider how AI is used when it comes to content.

I planned and wrote most of this article back in September 2023 and sat on it because I thought, who is going to believe me?

Now that it’s March 2024 and the SEO industry is facing a reckoning based partly on AI-generated content, people may be more receptive to considering better ways to integrate AI into the content generation workflow.

Google’s Website Cache Is Still Available (For Now) via @sejournal, @martinibuster

Google recently updated it’s Google Cache documentation on their website cache and in so doing indirectly created a reminder that the cache is still available for virtually any webpage that Google has indexed.

What Google Said About The Webpage Cache

What was reported about Google’s cache may have unintentionally left the impression that it was permanently and irretrievably gone. But that’s not yet the case.

Here’s SearchLiaison’s announcement on on Twitter:

“Hey, catching up. Yes, it’s been removed. I know, it’s sad. I’m sad too. It’s one of our oldest features. But it was meant for helping people access pages when way back, you often couldn’t depend on a page loading. These days, things have greatly improved. So, it was decided to retire it.

Personally, I hope that maybe we’ll add links to @internetarchive from where we had the cache link before, within About This Result. It’s such an amazing resource. For the information literacy goal of About The Result, I think it would also be a nice fit — allowing people to easily see how a page changed over time. No promises. We have to talk to them, see how it all might go — involves people well beyond me. But I think it would be nice all around.

As a reminder, anyone with a Search Console account can use URL Inspector to see what our crawler saw looking at their own page: https://support.google.com/webmasters/answer/9012289”

The cache is indeed gone from Google Search. But it’s still available as a search operator.

The reporting was correct that the cache was gone from Search but the part about its availability as a search operator got drowned out in the noise.

SearchLiaison was up front about the search operator.

His tweet continued:

“You’re going to see cache: go away in the near future, too.”

The “Cache:” Search Operator Is Still Working

Google recently updated their Search Central documentation on the cache: search operator to remove instructions for how to view the cache directly from the search results. But that’s it. There is no additional disclaimer that the cache: search operator is going away.

Google’s updated documentation removed references to the cache in search from two sections.

The documentation removed the following sentences:

“There are two ways to find the cached version of a page:
Search for cache: followed by the URL of the page, for example:
cache:https://example.com/your/page.html
Search for the URL, then click the 3 dots or arrow in the corner of the result to access a link to the cached version of the page.”

Google’s new documentation replaced the above paragraph with the following reworded passage:

“To find the cached version of a page, search for cache: followed by the URL of the page, for example:
cache:https://example.com/your/page.html”

The second change removed references to the cache in search from this passage (italicized part is removed):

“Most pages that Google indexed have a cached version, too. When a page doesn’t have a cached version, the previously mentioned methods for finding the cached version will fail…”

The above passage is replaced by the following one:

“Most pages that Google indexed have a cached version, too. When a page doesn’t have a cached version, using the cache: search operator to find the cached version will fail…”

Google’s SearchLiaison said that the cache: search operator was going away in the near future. The suggestion that Google might add a link to Internet Archive is not a useful replacement.

The reason it’s not useful is that the Google cache: search operator is useful for checking if competitor pages are indexed, not indexed or recently indexed – which is useful information.

The cache search operator is still so enjoy it while it lasts.

Read Google’s documentation about the cache: search operator:

cache: search operator