Ecommerce Management Advisors
Ecommerce Management Advisors
Google explains ‘not indexed’ status, advises improving content quality and server performance to enhance visibility in search results.
Google’s Gary Illyes recently highlighted a recurring SEO problem on LinkedIn, echoing concerns he’d previously voiced on a Google podcast.
The issue? URL parameters cause search engines difficulties when they’re crawling websites.
This problem is especially challenging for big sites and online stores. When different parameters are added to a URL, it can result in numerous unique web addresses that all lead to the same content.
This can impede search engines, reducing their efficiency in crawling and indexing sites properly.
In both the podcast and LinkedIn post, Illyes explains that URLs can accommodate infinite parameters, each creating a distinct URL even if they all point to the same content.
He writes:
“An interesting quirk of URLs is that you can add an infinite (I call BS) number of URL parameters to the URL path, and by that essentially forming new resources. The new URLs don’t have to map to different content on the server even, each new URL might just serve the same content as the parameter-less URL, yet they’re all distinct URLs. A good example for this is the cache busting URL parameter on JavaScript references: it doesn’t change the content, but it will force caches to refresh.”
He provided an example of how a simple URL like “/path/file” can expand to “/path/file?param1=a” and “/path/file?param1=a¶m2=b“, all potentially serving identical content.
“Each [is] a different URL, all the same content,” Illyes noted.
Search engines can sometimes find and try to crawl non-existent pages on your site, which Illyes calls “fake URLs.”
These can pop up due to things like poorly coded relative links. What starts as a normal-sized site with around 1,000 pages could balloon to a million phantom URLs.
This explosion of fake pages can cause serious problems. Search engine crawlers might hit your servers hard, trying to crawl all these non-existent pages.
This can overwhelm your server resources and potentially crash your site. Plus, it wastes the search engine’s crawl budget on useless pages instead of your content.
In the end, your pages might not get crawled and indexed properly, which could hurt your search rankings.
Illyes states:
“Sometimes you might create these new fake URLs accidentally, exploding your URL space from a balmy 1000 URLs to a scorching 1 million, exciting crawlers that in turn hammer your servers unexpectedly, melting pipes and whistles left and right. Bad relative links are one relatively common cause. But robotstxt is your friend in this case.”
The LinkedIn post didn’t specifically call out online stores, but the podcast discussion clarified that this issue is a big deal for ecommerce platforms.
These websites typically use URL parameters to handle product tracking, filtering, and sorting.
As a result, you might see several different URLs pointing to the same product page, with each URL variant representing color choices, size options, or where the customer came from.
Illyes consistently recommends using robots.txt to tackle this issue.
On the podcast, Illyes highlighted possible fixes, such as:
In the podcast discussion, Illyes touched on Google’s past attempts to address this issue, including the now-deprecated URL Parameters tool in Search Console.
This tool allowed websites to indicate which parameters were important and which could be ignored.
When asked on LinkedIn about potentially bringing back this tool, Illyes was skeptical about its practical effectiveness.
He stated, “In theory yes. in practice no,” explaining that the tool suffered from the same issues as robots.txt, namely that “people couldn’t for their dear life figure out how to manage their own parameters.”
This ongoing discussion from Google has several implications for SEO and web development:
Google is discussing URL parameter issues across multiple channels, which indicates a genuine concern for search quality.
For industry experts, staying informed on these technical aspects is essential for maintaining search visibility.
While Google works on solutions, proactive URL management and effective crawler guidance are recommended.
Image Credit: Lyna ™Strategy is the most misunderstood topic in the SEO world and likely the most impactful one. Just search for “SEO strategy” and notice that almost every article is about tactics. I get it. Tactics are sexier.
HOWEVER, focusing on tactics without a strategy increases your risk of:
Okay, the last one might not be true. But the others are. I’ve seen it many times. It’s ugly.
In an environment of aggressive algorithm updates, fewer clicks due to AI Overviews and more SERP real estate taken by forums, you cannot afford to take those risks. And yet, everyone does. I’m no exception: After spending the last two years crafting SEO strategies for companies like Ramp, Reddit, Nextdoor, and Hims, I realized the article that previously occupied this URL was too tactical – and completely rewrote it.
I’ve seen the difference between no and elegant strategy. Good SEO strategies embody a framework that seizes the opportunity of change, focuses on leverage, and beautifully stacks tactics in a logical structure.
To bring strategy to the forefront of SEO, I’m sharing with you:
Short definition: An SEO strategy defines how to overcome critical challenges by leveraging competitive advantages.
After years of not really getting it, I located the best definition of a business strategy in Richard Rummelt’s book “Good Strategy, Bad Strategy”. He explains that the kernel of strategy holds three parts:
A myriad of strong examples convinced me that this is the framework to go – and to apply to SEO. I’m rewording Rummelt’s framework to make it more colorful and approachable:
The SEO strategy kernel (Image Credit: Kevin Indig)Long definition: An SEO strategy clearly defines the Challenge a company must overcome through deep analysis. It concludes the direction in which the company needs to move to overcome that challenge in a framework called the Approach. Lastly, an SEO strategy defines specific Actions that must be taken to implement the Approach and overcome the Challenge. The Challenge remains constant, but the Approach can adjust, and Actions can change completely based on new information.
In his book “Originals”, Adam Grant even mentions that flexibility towards actions can be a strength:
Teams that evaluated their strategies at the midpoint were 80 percent more effective than teams that had the conversation at the start. […] This is one of the reasons that halftimes can be so influential in basketball and football: They allow coaches to intervene when teams are most amenable to new strategies.
You can tell a strategy is well thought-out based on four indicators:
Boost your skills with Growth Memo’s weekly expert insights. Subscribe for free!
The Challenge stands in the way of achieving your goals and embodies the “why” behind what you do. A strategy’s job is to find and solve the core challenge.
Defining the challenge is the most important part of a good SEO strategy because it’s the base for the direction and future actions. If you get this wrong, you waste a lot of time and money. To avoid failure, you need to invest in researching your market position, customers and competitors to understand the problems, their causes, and the ecosystems in which they occur.
Example: You aim to “win” the pet food SEO space but face established incumbents with more resources. After doing the research, you come to the conclusion that incumbents have outsized authority and relevance for pet food product names and category keywords.
The Approach is the “what” you do to overcome the core challenge. It’s a framework that sets the direction of your actions by exploiting your current advantages and exploring new ones. Getting the approach wrong means you lose focus and work on actions that don’t help you overcome the core challenge.
The Approach is an unambiguous list of the areas in which you invest and which you do not. Defining what you’re not going to do is equally as important as what you’re going to do because every strategy is a choice. That’s why my framework is called the 5 Choices. Without making hard choices, you likely don’t focus enough.
The Approach outlines your strategy risks, i.e., where the execution of your approach could go wrong. Every choice bears risk. By getting ahead of them, you minimize your chances of failing.
Critical: the approach needs to be differentiated. You need to do things differently (competitive advantage or asymmetry). You cannot expect to do the same things as your competitors and beat them. That’s just a way to end up in attrition warfare and obsession with operational efficiency. Differentiation creates greater value, prices and margins.
Sources of competitive advantages can be:
Example: To overcome the core challenge in the pet food space, you decide to leverage your strength of inhouse experts (say, you have a bunch of vets on staff). You invest in writing pet food guides instead of focusing on category and product landing pages. Traffic to the guides is funneled to landing pages where you sell your product.
Actions are “how” you implement the approach to overcome the core challenge. This is where it gets more tactical. But instead of picking “cool tricks” or “shiny objects”, your actions need to align with the approach and challenge. You know you’re there when your actions reinforce each other.
Example: Actions you could take to focus on pet food guides are publishing 100 pieces of content with detailed content briefs over the next 3 months, define a regular content refresh cadence and syndication to 10 industry magazines. You also create a video for each guide and position your site as a source of expert guidance from field practitioners.
Notice how you could have taken a different approach to solving the core challenge, like focusing on category and product landing pages. However, this approach would’ve likely not solved your core challenge because you know that strong incumbents already have a stronghold in that area. So, you need to do something different that plays into your strengths.
Instacart is a marketplace aggregator connecting buyers with supermarkets (read the full case study). Even though the company has been around since 2012, it encountered a defining moment when the COVID-19 pandemic broke out and consumers ordered groceries online.
Instacart’s approach was to capitalize on the trend as an aggregator instead of a supermarket chain. They tried to differentiate with inspirational editorial content but missed an opportunity to gain more visibility with recipes. Competitor Kroger regained the lead in 2022, and incumbent Costco is still far ahead.
Image Credit: Kevin IndigSubstack is a newsletter and blog hosting platform (read the case study) for journalists and writers.
Substack was able to catch up to WordPress in 2024 but is still far behind Medium, which had strong traffic declines between early 2021 and mid-2023.
Image Credit: Kevin IndigAdobe is a legendary software solution for designers and marketers (read the case study). After many years of strong growth from editorial content and lead-gen tools, organic traffic plateaued, and competitor Canva grabbed more traffic share.
Image Credit: Kevin IndigAdobe was able to recapture the top spot in organic traffic in 2024 after years of falling behind.
Non-SEO Example: Tesla
Tesla famously entered and captured the EV market with a luxury car instead of an affordable mass-market model. https://www.tesla.com/blog/secret-tesla-motors-master-plan-just-between-you-and-me Why?
From there, the company always planned to go down-market and build cheaper models. Today, Tesla has 50% market share.2
Simple and effective.
In my work as an inhouse leader and external advisor, I consistently come across four misunderstandings of SEO strategy:
1/ A strategy is not a set of tactics. As I explain in my free course Crafting a Winning SEO Strategy, “tactics are not defensible because they can be copied.” But for some reason, almost every article on the web about SEO strategy is all tactics and no strategy. Tactics are part of a strategy, but they are not the strategy.
I will never forget an executive at a renowned company saying, “This looks like we’re doing a bunch of stuff to me,” when I presented my “strategy”. The problem: I didn’t present the context (Challenge & Approach) in which the tactics make sense. And they didn’t. After taking a couple steps back, I re-evaluated my approach and presented a strategy that got us to win.
2/ A strategy is not a roadmap. A roadmap is a plan that defines which tactics are prioritized when and who executes them. Strategies lay out the playing field and how to win.
The following artifacts are plans, not strategies:
3/ A strategy is not a goal. The strategy explains how to achieve a goal, but it doesn’t stop there. And a strategy is certainly not a number. Strategies lay out the decisions that have to be made to achieve a goal and how to make them.
4/ A strategy is not operational efficiency. Tactics, roadmaps and goals can be part of a strategy, but operational efficiency is a way to run an organization. I get the excitement to work on operational efficiency because it’s impactful and actionable.
When you confuse operational efficiency with strategy, you ultimately end up in an attrition game because you try to do what everyone else does better. The problem is that you won’t be able to catch up to incumbents who are lightyears ahead. The way out is a differentiated strategy.
So many SEOs fall back to tactics because there is no good SEO strategy framework on the web. My goal with this article is to give you a framework that’s easy to understand and implement so you can build your own strategy. I call it The 5 Choices.
The 5 Choices (Image Credit: Kevin Indig)The 5 choices are a set of vital decisions you must make to land at a differentiated SEO strategy by carving out your Challenge, Approach and Actions.
A strategy isn’t worth its bits if it ends up in a Dropbox folder. It must be applied. And to be applied, it must make an impact. If SEO cannot impact the company meaningfully, you likely won’t get enough buy-in and funding to execute a differentiated strategy. Potential impact determines your Challenge.
Measure impact in revenue. For example, the potential or current revenue contribution from SEO or the ability of incremental revenue from SEO with lower ad budgets.
To diagnose the (potential) impact of SEO, see if projected traffic and revenue over the next 6-36 months are meaningful to the company.
Key questions:
Examples: High impact is when SEO accounts for 20% of the company’s revenue. Low impact is when search volume for key terms doesn’t translate into a meaningful opportunity.
The next question is whether the company is an SEO Integrator or Aggregator, based on who creates the content. If a company can scale SEO with user-generated content (UGC) or an inventory (e-commerce products, locations, or digital products), it’s an SEO aggregator. If a company creates the content “itself” with inhouse or freelance writers, it’s an SEO Integrator.
The difference is critical because each type scales SEO in its own way and has different levers in its arsenal, leading to distinct competitive advantages and approaches. Your type defines your Approach.
Key questions:
Examples: Instacart aggregates supermarkets and groceries, while Substack aggregates articles. Adobe is an integrator that built an aggregator arm with stock photos.
Early on, every company needs to focus on a few key user intents and topics to succeed in SEO. Over time, companies tend to fail in SEO when their domain carries too much content for topics they’re not yet known or an authority for. Topics and user intents can only expand with new products or features. Knowing which ones to focus on is a critical constraint for your Challenge and Approach.
Key questions:
To find key topics, look for which use cases best describe your product. Follow the jobs-to-be-done your product solves to find which user intents you can satisfy.
Examples: Airbnb targets people who search for activities and stays in cities (user intent) across core topics, such as short-term rentals, trips, and experiences. Adobe started as web design software and branched out into video and audio engineering, documents, marketing and e-commerce.
Knowing your advantage is critical so you can fully lean into it. Sometimes hard to find, every company has at least one advantage based on its positioning, resources and product/market fit that determines how a company could combat a problem (Approach).
One part of leaning into your strengths is being clear about what you won’t do. Boundaries create focus and can be turned into strengths. Companies win by leveraging strengths, not fixing weaknesses.
From Jim Collins’ book “Good to Great”:
The good-to-great companies did not focus principally on what to do to become great; they focused equally on what not to do and what to stop doing.
Key questions:
To find advantages, look for inflexible structures competitors have to defend. Every strategic choice is a commitment that a competitor can position themselves against.
Examples: In my deep dive into the company Course Hero, I describe how its main competitor, Studocu, chose to compete by focusing on international markets before tackling the US. That is a very distinct strategic choice based on Studocu’s competitive advantage, the fact that Course Hero has established itself in the US, and
There is no sustained impact without scale, whether you build more landing pages or hire more writers. Good strategies have force multipliers that compound returns over time with synergetic Actions.
Key questions:
To find scale mechanics, look for ways to create more high-quality pages on your domain and optimize your existing pages.
Examples: Local services aggregator Angi, for example, scales by consolidating its acquired companies into one domain and expanding editorial content with a large team.
Building your strategy is a 3 step process:
Research/analyze
Your goal is to develop a razor-sharp understanding of the problem(s) you’re facing and to decide which one is the key problem based on thorough research:
Document
Your strategy can be a written document or slide deck. The format itself is less important than compressing all the research and analysis into key paragraphs or slides.
Explain your reasoning in detail in the appendix or a separate document so that key partners understand where you’re coming from and why you come to your conclusions.
Communicate
The right way to communicate your strategy is with key partners first, ideally in person or virtually, and then with the broader company. Beat the drum on your strategy as often as you can with people at the forefront of execution. It takes a lot of repetition for the strategy to sink in, especially with larger companies.
Your strategy must be easy to understand. If it takes you 30 minutes and 100 slides to understand what the strategy is, you’re on the wrong track. Equally, if not every one of your employees can say what the strategy is, it’s likely not comprehensive enough.
First, you need clarity of the strategy for yourself, then you need to communicate it. – Bob Iger
The key to getting there is not to pick a simple strategy but to compress the hard work into its essence. A good example is how CEO Bob Iger simplified his strategy:
In Growth Memo Premium, I dissect the strategies of Nike, The New York Times, Redfin, IBM and other successful companies. Each succeeds with a differentiated SEO strategy that leads them to either dominate their industry or challenge incumbents.
A strong strategy aligns the whole organization toward a core challenge. It puts tactics into context and creates synergies between them, which means you’re getting more impact out of them than if you started with tactics and backed up into strategy.
I’ll leave you with these provocative questions:
This post was sponsored by SERPs. The opinions expressed in this article are the sponsor’s own.
Providing SEO services to businesses with numerous locations can be challenging without a refined process. Multi-location SEO can be a drain on efficiency, impacting your ability to take on bigger projects or more clients.
However, delivering high-quality, Google-compliant SEO at scale doesn’t have to be difficult.
The secret is applying just enough automation where it counts to complement the real human work and knowledge of your agency. One highly effective strategy is leveraging AI for geo-targeted landing page creation – streamlining repetitive tasks and allowing you to do more with fewer resources.
With the right approach, you can achieve impactful local SEO at scale, ensuring better outcomes for your clients and sustainable growth for your business.
Not sure how to start? SERPs offers geo page automation tools and customized support to help you integrate white-hat landing page automation into your local SEO workflows.
This article covers four ways to enhance your multi-location service packages using AI and automation with tools like SERPs.
Automation needs guardrails to ensure effective results and compliance with Google’s best practices. White-hat programmatic SEO requires planning, but this upfront work ensures a successful campaign that delivers the ROI your clients are looking for.
The first step is to develop a deep understanding of your client’s brand, ideal customer profiles (ICP), and current content strategy. This research gives you the keywords, emotional hooks, benefits, and features to guide the automation process.
This ensures that the landing pages your automated system creates are effective and also compliant with each brand’s style guide, voice, and marketing messaging.
When you use SERPs’ premium local SEO software, their team of experts will assist you by performing a full review of a domain of your choice. They will help you identify gaps in your clients’ strategy and website, as well as in your current SEO process.
Using this knowledge, you can build an SOP for all your current clients and future onboarding operations. A swift and insightful onboarding process will distinguish your agency from its competitors and set your campaigns up for success.
Before the algorithms generate new pages, develop high-quality seed content that they can draw from.
This is key to keeping the process white-hat and the outputs high-quality. Your team’s expertise shines during this stage because the better the seed content, the better the results of the campaign.
Think about how automated assets work in Google Ads. The AI tools take existing assets from a website and use them to generate new content to optimize ads. Your process will work similarly. The more you give the algorithms to work with, the better they’ll be able to optimize new pages.
The services SERPs provides are dedicated to helping agencies grow their local SEO programs and MRR.
They provide tools and strategies to streamline the production of geographically targeted landing pages. Your team provides the exceptional creative assets your clients already love you for.
This partnership of creativity and process leverages your expertise and knowledge of your clients to develop effective white-hat automation.
Once you’ve established guidelines and developed seed content, apply AI tools to create landing pages for each geolocation your client wants to target.
To ensure that the new landing pages follow the necessary guidelines, place specific variables on your seed pages. This directs the AI where to apply variable content and ensures that critical messaging remains consistent across all of the new geo pages.
You maintain control of the quality standards because the AI isn’t creating the bulk of the original content. The purpose of the software is to allow your team to execute necessary optimization tasks automatically.
This is the ideal combination of expert human touch and automated workflow. It improves your capacity to deliver results at scale while also improving outcomes for your clients.
The results speak for themselves. SERPs has a number of testimonials and case studies you can review attesting to the effectiveness of this approach. Combining expertise and automation in this way is safe and efficient. Your customers will love the results, and so will ranking algorithms.
Different communities have different needs. Ask your clients about particular communities and locations they want to focus on.
High-priority pages will benefit from any insights you can gather about the needs of local customers. In addition to any information your client has, you can use reviews, questions, and local demographics to refine the messaging of individual pages once they’ve been created.
Local knowledge takes landing pages a step further in meeting Google’s quality standards. Identify any priority locations early in the client’s campaign so you can note which landing pages may require an additional touch after creation.
The best use of local SEO automation is in combination with your skills and knowledge as a marketer. Premium service and efficiency with automation don’t have to be mutually exclusive. Providing exceptional local SEO pages at scale while reducing the impact on your team improves both your service offerings and your ROI on those services.
Effective local SEO at scale can be achieved through best practices that prioritize human insight, efficient SOPs, and judicious use of automation.
The best way to think about AI is as a force multiplier to your efficiency. You must start with research, insights, and content created by humans to ensure high-quality outputs and white hat processes.
Then, apply AI to quickly execute landing page creation tasks according to specific variables.
The power of creating exceptional geo-targeted landing pages at scale is in freeing your team to do more of their best work and fewer monotonous tasks.
Deliver optimized local SEO landing pages with high-quality assets and content with SERPs.com. Increase traffic and revenue for your clients with fewer resources and a higher ROI on your services.
Image Credits
Featured Image: Image by SERP’s. Used with permission.
Finding it tough to stay on top of AI’s impact on SEO?
You’re not the only one.
Google’s new AI overviews are changing how search works, and keeping up is more important than ever.
Check out our webinar on August 28, 2024: “AI Overviews Explained: Expert Embedding Techniques for SEO Success”
Market Brew is hosting this session to help you understand AI content and give you the tools to succeed in this new search environment.
AI overviews are changing how search engines show and organize content. To succeed in SEO, you need to understand how they work.
Save your seat on this webinar, as we cover:
Scott Stouffer, Co-Founder and CTO of Market Brew will also be showcasing Market Brew’s AI Overviews Visualizer, a tool that deconstructs AI overviews and offers an unprecedented look at how they’re curated.
This webinar is perfect for:
Following the presentation, Scott will host a live Q&A session.
This is your chance to clarify misconceptions and get expert advice on optimizing your content for AI overviews.
AI Overviews are reshaping SEO, and this shift is only accelerating. Join our August 28 webinar to futureproof your strategy.
Can’t attend live? Sign up anyway for the recording.
Get ready to supercharge your SEO with AI. Register today!
Google’s John Mueller explained on Reddit how to remove unwanted content from search results.
This came up when someone asked about getting rid of an old article about their arrest that kept showing up in Google searches.
The person was arrested for a minor offense in 2018, but a news article appears in Google searches years later.
Even though the case was settled, the article is still on the first page of results, and the person wants it removed.
What can they do? Here’s what Mueller advised.
Mueller explained that even though the news outlet said they “de-indexed” the article, this process isn’t always quick or simple.
He suggested a few ways to tackle the issue:
About the article still showing up in searches, Mueller said that even after de-indexing, it can take up to six months for a page to disappear from results:
“Regarding how long it takes to “see” a noindex, there’s no specific time, but it’s usually less than a few months. I think I’ve seen it take up to 6 months. They’re not kept in the index forever without being refreshed. If you use the public removal tool (for non-site-owners), Google will check the page fairly quickly (within a few days) and use that to confirm that the page has a noindex.”
He assured that pages don’t stay indexed forever without being rechecked.
Mueller mentioned that while some “hidden” de-indexing methods exist, they’re not common.
He recommended using Google’s public removal tool, which allows Google to recheck the page within days. This might speed things up if the news outlet has properly de-indexed the article.
Mueller stated:
“It’s really rare (I can’t think of any case in the last year or so) that someone would use a kind of “hidden” noindex; it’s complicated to set up & maintain. Most sites just use the visible robots meta tag for switching things to noindex, which you would be able to see yourself fairly quickly. If you use the removal tool, Google will also see any “more hidden” noindex settings.”
This advice gave the person a better grasp of their situation and possible next moves to deal with their lingering online content problem.
Mueller listed two main ways to get rid of content from search results:
If you own the site, Google removes the content on request.
For non-owners, Google does a few checks before taking anything down.
Mueller said using these tools won’t accidentally make your page show up more in searches.
He stated:
“The removal tool for site-owners has a help page titled “Removals and SafeSearch reports Tool”, subtitle “Temporarily block search results from your site, or manage SafeSearch filtering”. (Site-owner = the person running the website, in their Search Console account)
The public removal tool for non-site-owners is titled “Refresh Outdated Content tool” / subtitle: “Request an update to outdated content in Google Search results” (non-site-owner would be someone who doesn’t work on the website themselves, like you).
The site-owner tool will process a removal very quickly, without checking if the page is actually noindex or not. The assumption is that as the site-owner, you can block whatever you want. If they’re willing to do this for you, that’s the fastest way.
For non-site-owners, the tool will check multiple times to confirm that the page is removed, noindex, or appropriately changed. It won’t do anything until it has confirmed that, so there’s no harm in trying it. Neither of these tools will make a page more visible (SEOs would love that). The tools are also labeled as “temporary” removals – because if the page becomes indexable again, it can show up again in search.”
This shows how difficult it can be to manage what people see about you online.
While Google offers ways to remove old or unwanted articles, it can take a while, and sometimes, the publisher must cooperate.
Featured Image: tomeqs/Shutterstock
Google is currently addressing a separate issue affecting search rankings, unrelated to the August 2024 core update.
There’s an ongoing issue with ranking in Google Search that’s affecting a large number of search results. We’ve identified the root cause. This issue is unrelated to the ongoing core update rollout announced yesterday. We’re working on a fix. https://t.co/bxHy20KJJ0
— Google Search Central (@googlesearchc) August 16, 2024
Google has updated its guidance on core algorithm updates, providing more detailed recommendations for impacted websites.
The revised document, published alongside the August core update rollout, includes several additions and removals.
The most significant change includes two new sections: “Check if there’s a traffic drop in Search Console” and “Assessing a large drop in position.”
The “Check if there’s a traffic drop in Search Console” section provides step-by-step instructions for using Search Console to determine if a core update has affected a website.
The process involves:
The “Assessing a large drop in position” section offers guidance for websites that have experienced a significant ranking decline following a core update.
It recommends thoroughly evaluating the site’s content against Google’s quality guidelines, focusing on the pages most impacted by the update.
The updated document also includes a “Things to keep in mind when making changes” section, encouraging website owners to prioritize substantive, user-centric improvements rather than quick fixes.
It suggests that content deletion should be a last resort, indicating that removing content suggests it was created for search engines rather than users.
Another new section, “How long does it take to see an effect in Search results,” sets expectations for the time required to see ranking changes after making content improvements.
Google states that it may take several months for the full impact to be reflected, possibly requiring waiting until a future core update.
The document adds a closing paragraph noting that rankings can change even without website updates as new content emerges on the web.
Several sections from the previous version of the document have been removed or replaced in the update.
The paragraph stating that pages impacted by a core update “haven’t violated our spam policies” and comparing core updates to refreshing a movie list has been removed.
The “Assessing your own content” section has been replaced by the new “Assessing a large drop in position.”.
The “How long does it take to recover from a core update?” section no longer contains specific details about the timing and cadence of core updates and the factors influencing recovery time.
There’s a noticeable shift in tone and focus with this update.
While the previous guide explained the nature and purpose of core updates, the revised edition has more actionable guidance.
For example, the new sections related to Search Console provide clearer direction for identifying and addressing ranking drops.
Here’s a list of added and removed items in Google’s updated Core Algorithm Update Guidance.
Added:
Removed:
An archived version of Google’s previous core update guidance can be accessed via the Wayback Machine.
Featured Image: salarko/Shutterstock
Google was recently granted a patent on ranking web pages, which may offer insights into how AI Overviews ranks content. The patent describes a method for ranking pages based on what a user might be interested in next.
The name of the patent is Contextual Estimation Of Link Information Gain, it was filed in 2018 and granted in June 2024. It’s about calculating a ranking score called Information Gain that is used to rank a second set of web pages that are likely to be of interest to a user as a slightly different follow-up topic related to a previous question.
The patent starts with general descriptions then adds layers of specifics over the course of paragraphs. An analogy can be that it’s like a pizza. It starts out as a mozzarella pizza, then they add mushrooms, so now it’s a mushroom pizza. Then they add onions, so now it’s a mushroom and onion pizza. There are layers of specifics that build up to the entire context.
So if you read just one section of it, it’s easy to say, “It’s clearly a mushroom pizza” and be completely mistaken about what it really is.
There are layers of context but what it’s building up to is:
A couple of months ago I read a comment on social media asserting that “Information Gain” was a significant factor in a recent Google core algorithm update. That mention surprised me because I’d never heard of information gain before. I asked some SEO friends about it and they’d never heard of it either.
What the person on social media had asserted was something like Google was using an “Information Gain” score to boost the ranking of web pages that had more information than other web pages. So the idea was that it was important to create pages that have more information than other pages, something along those lines.
So I read the patent and discovered that “Information Gain” is not about ranking pages with more information than other pages. It’s really about something that is more profound for SEO because it might help to understand one dimension of how AI Overviews might rank web pages.
What the information gain patent is really about is even more interesting because it may give an indication of how AI Overviews (AIO) ranks web pages that a user might be interested next. It’s sort of like introducing personalization by anticipating what a user will be interested in next.
The patent describes a scenario where a user makes a search query and the automated assistant or chatbot provides an answer that’s relevant to the question. The information gain scoring system works in the background to rank a second set of web pages that are relevant to a what the user might be interested in next. It’s a new dimension in how web pages are ranked.
There are multiple versions of the Information Gain patent dating from 2018 to 2024. The first version is similar to the last version with the most significant difference being the addition of chatbots as a context for where the information gain invention is used.
The patent uses the phrase “automated assistant” 69 times and uses the phrase “search engine” only 25 times. Like with AI Overviews, search engines do play a role in this patent but it’s generally in the context of automated assistants.
As will become evident, there is nothing to suggest that a web page containing more information than the competition is likelier to be ranked higher in the organic search results. That’s not what this patent talks about.
All versions of the patent describe the presentation of search results within the context of an automated assistant and natural language question answering. The patent starts with a general description and progressively becomes more specific. This is a feature of patents in that they apply for protection for the widest contexts in which the invention can be used and become progressively specific.
The entire first section (the Abstract) doesn’t even mention web pages or links. It’s just about the information gain score within a very general context:
“An information gain score for a given document is indicative of additional information that is included in the document beyond information contained in documents that were previously viewed by the user.”
That is a nutshell description of the patent, with the key insight being that the information gain scoring happens on pages after the user has seen the first search results.
The second paragraph in the section titled “Background” is slightly more specific and adds an additional layer of context for the invention because it mentions links. Specifically, it’s about a user that makes a search query and receives links to search results – no information gain score calculated yet.
The Background section says:
“For example, a user may submit a search request and be provided with a set of documents and/or links to documents that are responsive to the submitted search request.”
The next part builds on top of a user having made a search query:
“Also, for example, a user may be provided with a document based on identified interests of the user, previously viewed documents of the user, and/or other criteria that may be utilized to identify and provide a document of interest. Information from the documents may be provided via, for example, an automated assistant and/or as results to a search engine. Further, information from the documents may be provided to the user in response to a search request and/or may be automatically served to the user based on continued searching after the user has ended a search session.”
That last sentence is poorly worded.
Here’s the original sentence:
“Further, information from the documents may be provided to the user in response to a search request and/or may be automatically served to the user based on continued searching after the user has ended a search session.”
Here’s how it makes more sense:
“Further, information from the documents may be provided to the user… based on continued searching after the user has ended a search session.”
The information provided to the user is “in response to a search request and/or may be automatically served to the user”
It’s a little clearer if you put parentheses around it:
Further, information from the documents may be provided to the user (in response to a search request and/or may be automatically served to the user) based on continued searching after the user has ended a search session.
Takeaways:
The patent next adds an additional layer of context and specificity by mentioning how chatbots can “extract” an answer from a web page (“document”) and show that as an answer. This is about showing a summary that contains the answer, kind of like featured snippets, but within the context of a chatbot.
The patent explains:
“In some cases, a subset of information may be extracted from the document for presentation to the user. For example, when a user engages in a spoken human-to-computer dialog with an automated assistant software process (also referred to as “chatbots,” “interactive personal assistants,” “intelligent personal assistants,” “personal voice assistants,” “conversational agents,” “virtual assistants,” etc.), the automated assistant may perform various types of processing to extract salient information from a document, so that the automated assistant can present the information in an abbreviated form.
As another example, some search engines will provide summary information from one or more responsive and/or relevant documents, in addition to or instead of links to responsive and/or relevant documents, in response to a user’s search query.”
The last sentence sounds like it’s describing something that’s like a featured snippet or like AI Overviews where it provides a summary. The sentence is very general and ambiguous because it uses “and/or” and “in addition to or instead of” and isn’t as specific as the preceding sentences. It’s an example of a patent being general for legal reasons.
The next section is called the Summary and it goes into more details about how the Information Gain score represents how likely the user will be interested in the next set of documents. It’s not about ranking search results, it’s about ranking the next set of search results (based on a related topic).
It states:
“An information gain score for a given document is indicative of additional information that is included in the given document beyond information contained in other documents that were already presented to the user.”
It then talks about presenting the web page in a browser, audibly reading the relevant part of the document or audibly/visually presenting a summary of the document (“audibly/visually presenting salient information extracted from the document to the user, etc.”)
But the part that’s really interesting is when it next explains using a topic of the web page as a representation of the the content, which is used to calculate the information gain score.
It describes many different ways of extracting the representation of what the page is about. But what’s important is that it’s describes calculating the Information Gain score based on a representation of what the content is about, like the topic.
“In some implementations, information gain scores may be determined for one or more documents by applying data indicative of the documents, such as their entire contents, salient extracted information, a semantic representation (e.g., an embedding, a feature vector, a bag-of-words representation, a histogram generated from words/phrases in the document, etc.) across a machine learning model to generate an information gain score.”
The patent goes on to describe ranking a first set of documents and using the Information Gain scores to rank additional sets of documents that anticipate follow up questions or a progression within a dialog of what the user is interested in.
The automated assistant can in some implementations query a search engine and then apply the Information Gain rankings to the multiple sets of search results (that are relevant to related search queries).
There are multiple variations of doing the same thing but in general terms this is what it describes:
“Based on the information gain scores, information contained in one or more of the new documents may be selectively provided to the user in a manner that reflects the likely information gain that can be attained by the user if the user were to be presented information from the selected documents.”
All versions of the patent share general similarities over which more specifics are layered in over time (like adding onions to a mushroom pizza). The following are the baseline of what all the versions have in common.
All versions of the patent describe applying the information gain score to a second set of documents that have additional information beyond the first set of documents. Obviously, there is no criteria or information to guess what the user is going search for when they start a search session. So information gain scores are not applied to the first search results.
Examples of passages that are the same for all versions:
- A second set of documents is identified that is also related to the topic of the first set of documents but that have not yet been viewed by the user.
- For each new document in the second set of documents, an information gain score is determined that is indicative of, for the new document, whether the new document includes information that was not contained in the documents of the first set of documents…
All four versions of the patent refer to automated assistants that show search results in response to natural language queries.
The 2018 and 2023 versions of the patent both mention search engines 25 times. The 2o18 version mentions “automated assistant” 74 times and the latest version mentions it 69 times.
They all make references to “conversational agents,” “interactive personal assistants,” “intelligent personal assistants,” “personal voice assistants,” and “virtual assistants.”
It’s clear that the emphasis of the patent is on automated assistants, not the organic search results.
Note: In everyday language we use the word dialogue. In computing they the spell it dialog.
All versions of the patents refer to a way of interacting with the system in the form of a dialog, specifically a dialog turn. A dialog turn is the back and forth that happens when a user asks a question using natural language, receives an answer and then asks a follow up question or another question altogether. This can be natural language in text, text to speech (TTS), or audible.
The main aspect the patents have in common is the back and forth in what is called a “dialog turn.” All versions of the patent have this as a context.
Here’s an example of how the dialog turn works:
“Automated assistant client 106 and remote automated assistant 115 can process natural language input of a user and provide responses in the form of a dialog that includes one or more dialog turns. A dialog turn may include, for instance, user-provided natural language input and a response to natural language input by the automated assistant.
Thus, a dialog between the user and the automated assistant can be generated that allows the user to interact with the automated assistant …in a conversational manner.”
The main feature of the patent is to improve the user experience by understanding the additional value that a new document provides compared to documents that a user has already seen. This additional value is what is meant by the phrase Information Gain.
There are multiple ways that information gain is useful and one of the ways that all versions of the patent describes is in the context of an audio response and how a long-winded audio response is not good, including in a TTS (text to speech) context).
The patent explains the problem of a long-winded response:
“…and so the user may wait for substantially all of the response to be output before proceeding. In comparison with reading, the user is able to receive the audio information passively, however, the time taken to output is longer and there is a reduced ability to scan or scroll/skip through the information.”
The patent then explains how information gain can speed up answers by eliminating redundant (repetitive) answers or if the answer isn’t enough and forces the user into another dialog turn.
This part of the patent refers to the information density of a section in a web page, a section that answers the question with the least amount of words. Information density is about how “accurate,” “concise,” and “relevant”‘ the answer is for relevance and avoiding repetitiveness. Information density is important for audio/spoken answers.
This is what the patent says:
“As such, it is important in the context of an audio output that the output information is relevant, accurate and concise, in order to avoid an unnecessarily long output, a redundant output, or an extra dialog turn.
The information density of the output information becomes particularly important in improving the efficiency of a dialog session. Techniques described herein address these issues by reducing and/or eliminating presentation of information a user has already been provided, including in the audio human-to-computer dialog context.”
The idea of “information density” is important in a general sense because it communicates better for users but it’s probably extra important in the context of being shown in chatbot search results, whether it’s spoken or not. Google AI Overviews shows snippets from a web page but maybe more importantly, communicating in a concise manner is the best way to be on topic and make it easy for a search engine to understand content.
All versions of the Information Gain patent are clear that the invention is not in the context of organic search results. It’s explicitly within the context of ranking web pages within a natural language interface of an automated assistant and an AI chatbot.
However, there is a part of the patent that describes a way of showing users with the second set of results within a “search results interface.” The scenario is that the user sees an answer and then is interested in a related topic. The second set of ranked web pages are shown in a “search results interface.”
The patent explains:
“In some implementations, one or more of the new documents of the second set may be presented in a manner that is selected based on the information gain stores. For example, one or more of the new documents can be rendered as part of a search results interface that is presented to the user in response to a query that includes the topic of the documents, such as references to one or more documents. In some implementations, these search results may be ranked at least in part based on their respective information gain scores.”
…The user can then select one of the references and information contained in the particular document can be presented to the user. Subsequently, the user may return to the search results and the references to the document may again be provided to the user but updated based on new information gain scores for the documents that are referenced.
In some implementations, the references may be reranked and/or one or more documents may be excluded (or significantly demoted) from the search results based on the new information gain scores that were determined based on the document that was already viewed by the user.”
What is a search results interface? I think it’s just an interface that shows search results.
Let’s pause here to underline that it should be clear at this point that the patent is not about ranking web pages that are comprehensive about a topic. The overall context of the invention is showing documents within an automated assistant.
A search results interface is just an interface, it’s never described as being organic search results, it’s just an interface.
There’s more that is the same across all versions of the patent but the above are the important general outlines and context of it.
The claims section is where the scope of the actual invention is described and for which they are seeking legal protection over. It is mainly focused on the invention and less so on the context. Thus, there is no mention of a search engines, automated assistants, audible responses, or TTS (text to speech) within the Claims section. What remains is the context of search results interface which presumably covers all of the contexts.
It starts out by outlining the context of the invention. This context is receiving a query, identifying the topic, and ranking a first group of relevant web pages (documents) and selecting at least one of them as being relevant and either showing the document or communicating the information from the document (like a summary).
“1. A method implemented using one or more processors, comprising: receiving a query from a user, wherein the query includes a topic; identifying a first set of documents that are responsive to the query, wherein the documents of the set of documents are ranked, and wherein a ranking of a given document of the first set of documents is indicative of relevancy of information included in the given document to the topic; selecting, based on the rankings and from the documents of the first set of documents, a most relevant document providing at least a portion of the information from the most relevant document to the user;”
Then what immediately follows is the part about ranking a second set of documents that contain additional information. This second set of documents is ranked using the information gain scores to show more information after showing a relevant document from the first group.
This is how it explains it:
“…in response to providing the most relevant document to the user, receiving a request from the user for additional information related to the topic; identifying a second set of documents, wherein the second set of documents includes at one or more of the documents of the first set of documents and does not include the most relevant document; determining, for each document of the second set, an information gain score, wherein the information gain score for a respective document of the second set is based on a quantity of new information included in the respective document of the second set that differs from information included in the most relevant document; ranking the second set of documents based on the information gain scores; and causing at least a portion of the information from one or more of the documents of the second set of documents to be presented to the user, wherein the information is presented based on the information gain scores.”
The rest of the claims section contains granular details about the concept of Information Gain, which is a ranking of documents based on what the user already has seen and represents a related topic that the user may be interested in. The purpose of these details is to lock them in for legal protection as part of the invention.
Here’s an example:
The method of claim 1, wherein identifying the first set comprises:
causing to be rendered, as part of a search results interface that is presented to the user in response to a previous query that includes the topic, references to one or more documents of the first set;
receiving user input that that indicates selection of one of the references to a particular document of the first set from the search results interface, wherein at least part of the particular document is provided to the user in response to the selection;
To make an analogy, it’s describing how to make the pizza dough, clean and cut the mushrooms, etc. It’s not important for our purposes to understand it as much as the general view of what the patent is about.
An opinion was shared on social media that this patent has something to do with ranking web pages in the organic search results, I saw it, read the patent and discovered that’s not how the patent works. It’s a good patent and it’s important to correctly understand it. I analyzed multiple versions of the patent to see what they had in common and what was different.
A careful reading of the patent shows that it is clearly focused on anticipating what the user may want to see based on what they have already seen. To accomplish this the patent describes the use of an Information Gain score for ranking web pages that are on topics that are related to the first search query but not specifically relevant to that first query.
The context of the invention is generally automated assistants, including chatbots. A search engine could be used as part of finding relevant documents but the context is not solely an organic search engine.
This patent could be applicable to the context of AI Overviews. I would not limit the context to AI Overviews as there are additional contexts such as spoken language in which Information Gain scoring could apply. Could it apply in additional contexts like Featured Snippets? The patent itself is not explicit about that.
Read the latest version of Information Gain patent:
Contextual estimation of link information gain
Featured Image by Shutterstock/Khosro
Google’s August 2024 core update targets low-value SEO content, promoting quality websites. Full rollout expected within a month.