Google Shows How To Check Passage Indexing via @sejournal, @martinibuster

Google’s John Mueller was asked how many megabytes of HTML Googlebot crawls per page. The question was whether Googlebot indexes two megabytes (MB) or fifteen megabytes of data. Mueller’s answer minimized the technical aspect of the question and went straight to the heart of the issue, which is really about how much content is indexed.

GoogleBot And Other Bots

In the middle of an ongoing discussion in Bluesky someone revived the question about whether Googlebot crawls and indexes 2 or 15 megabytes of data.

They posted:

“Hope you got whatever made you run 🙂

It would be super useful to have more precisions, and real-life examples like “My page is X Mb long, it gets cut after X Mb, it also loads resource A: 15Kb, resource B: 3Mb, resource B is not fully loaded, but resource A is because 15Kb < 2Mb”.”

Panic About 2 Megabyte Limit Is Overblown

Mueller said that it’s not necessary to weigh bytes and implied that what’s ultimately important isn’t about constraining how many bytes are on a page but rather whether or not important passages are indexed.

Furthermore, Mueller said that it is rare that a site exceeds two megabytes of HTML, dismissing the idea that it’s possible that a website’s content might not get indexed because it’s too big.

He also said that Googlebot isn’t the only bot that crawls a web page, apparently to explain why 2 megabytes and 15 megabytes aren’t limiting factors. Google publishes a list of all the crawlers they use for various purposes.

How To Check If Content Passages Are Indexed

Lastly, Mueller’s response confirmed a simple way to check whether or not important passages are indexed.

Mueller answered:

“Google has a lot of crawlers, which is why we split it. It’s extremely rare that sites run into issues in this regard, 2MB of HTML (for those focusing on Googlebot) is quite a bit. The way I usually check is to search for an important quote further down on a page – usually no need to weigh bytes.”

Passages For Ranking

People have short attention spans except when they’re reading about a topic that they are passionate about. That’s when a comprehensive article may come in handy for those readers who really want to take a deep dive to learn more.

From an SEO perspective, I can understand why some may feel that a comprehensive article might not be ideal for ranking if a document provides deep coverage of multiple topics, any one of which could be a standalone article.

A publisher or an SEO needs to step back and assess whether a user is satisfied with deep coverage of a topic or whether a deeper treatment of it is needed by users. There are also different levels of comprehensiveness, one with granular details and another with an overview-level of coverage of details, with links to deeper coverage.

In other words, sometimes users require a view of the forest and sometimes they require a view of the trees.

Google has long been able to rank document passages with their passage ranking algorithms. Ultimately, in my opinion, it really comes down to what is useful to users and is likely to result in a higher level of user satisfaction.

If comprehensive topic coverage excites people and makes them passionate enough about to share it with other people then that is a win.

If comprehensive coverage isn’t useful for that specific topic then it may be better to split the content into shorter coverage that better aligns with the reasons why people are coming to that page to read about that topic.

Takeaways

While most of these takeaways aren’t represented in Mueller’s response, they do in my opinion represent good practices for SEO.

  • HTML size limits belie a concern for deeper questions about content length and indexing visibility
  • Megabyte thresholds are rarely a practical constraint for real-world pages
  • Counting bytes is less useful than verifying whether content actually appears in search
  • Searching for distinctive passages is a practical way to confirm indexing
  • Comprehensiveness should be driven by user intent, not crawl assumptions
  • Content usefulness and clarity matter more than document size
  • User satisfaction remains the deciding factor in content performance

Concern over how many megabytes are a hard crawl limit for Googlebot reflect uncertainty about whether important content in a long document is being indexed and is available to rank in search. Focusing on megabytes shifts attention away from the real issues SEOs should be focusing on, which is whether the topic coverage depth best serves a user’s needs.

Mueller’s response reinforces the point that web pages that are too big to be indexed are uncommon, and fixed byte limits are not a constraint that SEOs should be concerned about.

In my opinion, SEOs and publishers will probably have better search coverage by shifting their focus away from optimizing for assumed crawl limits and instead focus on user content consumption limits.

But if a publisher or SEO is concerned about whether a passage near the end of a document is indexed, there is an easy way to check the status by simply doing a search for an exact match for that passage.

Comprehensive topic coverage is not automatically a ranking problem, and it not always the best (or worst) approach. HTML size is not really a concern unless it starts impacting page speed. What matters is whether content is clear, relevant, and useful to the intended audience at the precise levels of granularity that serves the user’s purposes.

Featured Image by Shutterstock/Krakenimages.com

Google Releases Discover-Focused Core Update via @sejournal, @MattGSouthern
  • Google has launched a core update specifically for Discover, rather than Search more broadly.
  • The February Discover core update began Feb. 5 for English-language users in the U.S., with plans to expand to other countries and languages.
  • Google says the rollout may take up to two weeks.

Google has started a Discover core update. The rollout may take up to two weeks, with expansion to more countries and languages later.

The Shift From Search Sessions To Decision Sessions via @sejournal, @DuaneForrester

This one started with a question from Adorján-Csaba Demeter, a subscriber in Romania, who asked how big the behavior change could be after Google’s AI Mode Personal Search launch, and it pushed me to think past the product announcement and into the habit shift underneath it.

AI changing search is a foregone conclusion. The real story is what happens to people when search stops acting like a library and starts acting like a helper that knows what you meant, what you like, and what you have coming up next.

When effort drops, behavior changes first. Then business models change. Then the web scrambles to catch up.

Image Credit: Duane Forrester

What Google Actually Changed

Google did not just add another AI layer to results. It moved AI Mode from “answer from the web” toward “answer from the web plus your life,” starting with opt-in connections to Gmail and Google Photos for AI Pro and AI Ultra subscribers in the U.S., delivered as a Labs experiment.

That detail matters because it tells you what Google thinks the next battleground is.

Not faster answers, but stickier habits.

When the system can read your hotel confirmation in Gmail, it can plan. When it can see the kinds of trips you take in Photos, it can recommend. You stop doing the work of explaining context. You start delegating outcomes.

That is a bet on human behavior.

The three behavior shifts that will most likely follow, in order, are:

1. People ask more questions, and they ask harder questions.

Google already sees this pattern with AI Overviews. In major markets like the U.S. and India, Google says AI Overviews drive over a 10% increase in usage for the types of queries that show them. That is a habit signal, not a satisfaction claim.

When people believe the system will do more for them, they return more often, and they push further. Queries get longer. They get more specific. They get more outcome-oriented. People stop asking “what is” and start asking “what should I do.”

Personal context amplifies that shift. If the system already knows your reservations, your preferences, and your recent activity, the user has less friction and more confidence. That increases question volume.

2. Sessions end sooner, and fewer decisions happen on websites.

Here’s the part businesses need to internalize. AI does not just reduce clicks. It compresses the journey and ends sessions earlier.

Pew’s browsing-panel study found that when an AI summary appeared, users clicked a traditional search result in 8% of visits versus 15% when there was no AI summary. Pew also found users were more likely to end their browsing session after a page with an AI summary, 26% versus 16% without.

3. People shift from browsing to delegating.

This is where behavior becomes durable. Traditional search trained people to open tabs, compare sources, build their own plan, then act. AI Mode personalizes the plan inside search itself. It turns “find me information” into “help me decide.” If the system can use your life context, it can do the assembly work you used to do manually.

That is the transition from search sessions to decision sessions. A search session ends when you find information. A decision session ends when you have a recommended next step and you are ready to act.

Adoption Will Be Real, And Uneven, For A Simple Reason

People like convenience, but they do not always like the feeling of being summarized.

Pew found that among Americans who have seen AI summaries in search results, only one in five say they find them extremely or very useful. Most say somewhat useful, and 28% say not too or not at all useful.

Low-stakes categories will move fastest because the cost of being wrong is low. High-stakes categories will move slower because trust and liability show up quickly, even when the convenience is obvious.

Even with mixed sentiment, usage is already going mainstream. Deloitte’s 2025 Connected Consumer survey found 53% of surveyed consumers are either experimenting with gen AI or using it regularly, up from 38% in 2024.

The behavior change is already underway, and I think Google is trying to capture it inside its existing habit loop.

What This Does To Businesses, Even If Your SEO Is Perfect

This is where most teams get stuck. They see AI Mode and AI summaries and assume it is “just another ranking change.” It is not. It is a consumer behavior change that reshapes the economics of discovery. The shift is subtle at first, then it hits you all at once, because it changes what people consider a completed search experience.

When sessions complete in the answer layer, classic top-of-funnel traffic becomes less reliable, even if your rankings hold. The competitive line shifts to inclusion: being referenced, cited, recommended, or selected as the next step inside the plan the system generates.

To win there, build for next-step intent. Most marketing content assumes the user will land on your site and then decide. AI compresses that journey, so your content has to carry options, tradeoffs, and a clear “what to do next,” in a form that survives summarization.

Vertical Impacts, Where Behavior Shifts First

Healthcare

People already use search as a first stop for health. The Annenberg Public Policy Center found that most (79%) U.S. adults say they’re likely to look online for the answer to a question about a health symptom or condition.

And the way they search is predictable. A 2025 JMIR survey study found participants most often sought information on health conditions, 90.2%, and medication info came next, 60.3%.

As the answer layer feels more confident, people will use it for triage and next steps. It will influence which clinic they choose and how quickly they escalate a concern.

Healthcare businesses should expect:

  1. Less website traffic for broad informational topics, and more pressure on “what do I do next” moments.
  2. Increased competition to be the cited and trusted source inside AI answers.
  3. Higher stakes for accuracy and clarity, because summarization can remove nuance.

There is also a revealing warning signal here. A study of health-related AI Overviews citations, found YouTube was the single most cited source, accounting for 4.43% of citations in that dataset.

That is not an argument against AI. It is a reminder that citation sources do not automatically align with medical rigor. Businesses in healthcare need to make their evidence, authorship, and care pathways machine-readable and unambiguous.

Financial Services

Finance is already living in an “assistant” world, and that matters because it shows how quickly consumers accept delegated help when it saves effort.

Bank of America reports that Erica (their consumer AI assistant) has surpassed 3.2 billion client interactions since its 2018 launch, and clients now interact with Erica more than 2 million times per day.

That is behavior change at scale.

Meanwhile, consumers are increasingly willing to use AI for financial advice and information. ABA Banking Journal reported in September 2025 that 51% of respondents said they turn to AI to get financial advice or information, and another 27% said they are considering it.

Now when we connect the dots…

If AI Mode personalizes search around a user’s life context, financial decision-making gets pulled earlier into the assistant layer. Budgeting questions, product comparisons, “should I refinance,” “how much house can I afford,” and “what happens if I miss a payment” all become conversational.

Financial services businesses should expect:

  1. Increased competition for being the recommended next step, not just being discoverable.
  2. More pressure to publish clear, plain-language product explanations that survive summarization.
  3. A sharper separation between low-stakes guidance and regulated advice, with trust and compliance becoming part of how content gets used.

Retail And Ecommerce

Retail gets hit hard because the classic behavior pattern is tab sprawl, and AI collapses it into a shortlist.

Retail businesses should expect:

  1. Fewer browsing sessions that start with generic research and end on a product page.
  2. More “shortlist behavior,” where the system presents a handful of options and the user picks.
  3. Higher importance for product data that can be summarized cleanly, including dimensions, compatibility, return policies, and warranty terms.

If your differentiation lives in fluffy copy, it dies in the summary. If it lives in measurable attributes, verified reviews, and clear tradeoffs, it survives.

Local Services

Local services are where this gets practical fast. People search when something broke, they need help now, and they do not want homework.

AI Mode personal context will steer choices based on urgency, location, constraints, and preferences. That means “best next step” routing becomes default behavior.

Local businesses should expect:

  1. Less opportunity to win by content volume alone.
  2. More emphasis on entity clarity, service area accuracy, availability, pricing ranges, and proof of credibility.
  3. A rise in “invisible funnel” decisions, where the customer shows up ready to book because the plan already happened elsewhere.

What You Can Do Today, Without Waiting For The Dust To Settle

For Consumers

1. Decide where you want personalization, and where you do not. Personal AI is a trade. You get convenience, but you give context. Make that choice deliberately.

2. Use AI for options, then verify what has consequences. Health, money, legal, and safety decisions deserve a second look. If an answer influences a purchase, a medical step, or a contract, capture the source and key details so convenience does not erase accountability.

For Businesses

1. Stop treating clicks as the only signal that matters. Clicks will drop in many query classes, and sessions will end sooner. Measure presence in answers, citations, recommendations, and downstream conversions that happen after exposure.

2. Rebuild your content around next-step intent. Take your highest value pages and rewrite them for decision completion. Clear options. Clear tradeoffs. Clear “what to do next.”

3. Make your entity impossible to misunderstand. Clean organization signals, consistent naming, authoritative profiles, accurate locations, and structured data where relevant. When the machine layer tries to explain who you are, make it easy.

4. Publish proof, not fluff. In high-stakes verticals, show your sources, your credentials, your policies, and your constraints. AI can compress text, but it still needs real signals to anchor trust.

The Competitive Forecast, Google Versus The Rest

If AI Mode personal search takes off, the winners will not be determined by model quality alone. Distribution and habit will do most of the work.

Scenario one, Google accelerates

Google’s biggest advantage is not that it can build an assistant. It is that it can place the assistant inside a habit billions of people already have. (Android + Siri) It already sees increased usage when AI Overviews appear, over 10% in major markets for those query types.

If Google can move Personal Intelligence from paid opt-in into broader availability, and expand the connected sources beyond Gmail and Photos, it can turn search into a daily operating layer for planning and decisions. That is a habit engine.

Scenario two, the market stays plural

ChatGPT and other assistants will continue to grow because they do not live only in “search.” They live in work, writing, learning, and deep tasks. Many users will keep separate habits, one for web discovery, another for assistant workflows, at least for a while.

In a plural market, businesses must optimize for multiple answer layers, not just Google.

What To Watch In 2026

  1. Whether Google keeps Personal Intelligence as a paid feature or uses it as a default habit builder.
  2. Whether connected context expands, and which sources get added next.
  3. Whether user sentiment shifts from lukewarm to reliant or stays mixed as Pew found.
  4. How quickly session compression shows up by vertical, since that will reveal where business disruption hits first.

The Takeaway

The change to watch is not that AI can answer questions. That part is already here, and it will keep improving. The real change is that people will stop doing the assembly work they have always done in search. They will ask more, browse less, and increasingly accept plans that arrive pre-built, because it feels faster and it feels complete. Habits will change.

When that happens, power moves upward into the answer layer. Competition shifts from who ranks to who gets included, because inclusion is what influences the decision before a user ever lands on your site. The web does not disappear, but its role changes. It becomes the dependency that feeds answers, not the destination where discovery naturally occurs.

If you run a business, you cannot pause this shift. You can adapt. Build for decision completion. Make your proof easy to carry forward so it survives summarization and still earns trust. Measure what matters when the click often disappears.

More Resources:


This post was originally published on Duane Forrester Decodes.


Featured Image: Collagery/Shutterstock

Google Search Hits $63B, Details AI Mode Ad Tests via @sejournal, @MattGSouthern

Alphabet reported Q4 2025 revenue of $113.8 billion, beating Wall Street estimates and marking the company’s first year above $400 billion in annual revenue. Google Search grew 17% to $63.07 billion.

On the earnings call, the company revealed how it plans to monetize AI Mode and shared new data on how AI is changing search behavior.

What’s Happening

Google Search and other advertising revenue hit $63.07 billion, up 17% from $54.03 billion in Q4 2024. Search growth accelerated through 2025, rising from 10% in Q1 to 12% in Q2 to 15% in Q3 and 17% in Q4.

CEO Sundar Pichai said Search had more usage in Q4 than ever before. He attributed the growth to AI features changing how people search.

Pichai said on the call:

“Once people start using these new experiences, they use them more. In the US, we saw daily AI Mode queries per user double since launch.”

Queries in AI Mode are three times longer than traditional searches, and a “significant portion” lead to follow-up questions.

AI Mode Monetization Tests

Chief Business Officer Philipp Schindler said Google is “in the early stages of experimenting with AI Mode monetization, like testing ads below the AI response, with more underway.”

On Direct Offers, a new pilot program, Schindler said:

“We announced Direct Offers, a new Google Ads pilot, which will allow advertisers to show exclusive offers for shoppers who are ready to buy, directly in AI Mode.”

Google also plans to launch checkout directly within AI Mode from select merchants.

Schindler said the longer AI Mode queries are creating new ad inventory. Gemini’s understanding of intent “has increased our ability to deliver ads on longer, more complex searches that were previously challenging to monetize.”

YouTube Miss Explained

YouTube ad revenue reached $11.38 billion, up 9% but below the $11.84 billion analysts expected.

Schindler attributed the miss to election ad lapping from Q4 2024:

“On the brand side, as an ad share, the largest factor negatively impacting the year-over-year growth rate was lapping the strong spend on U.S. elections.”

He also noted that subscription growth can reduce ad revenue. When users switch to YouTube Premium, it hurts ad revenue but helps the overall business.

What Else Happened

Google Cloud revenue jumped 48% to $17.66 billion. Alphabet plans to spend $175 billion to $185 billion on capital expenditures in 2026, nearly double its 2025 spending. That suggests more AI features coming to Search and other products.

Why This Matters

Looking back a year ago at Q4 2024 results, Search grew 12%. By Q1 2025, AI Overviews reached 1.5 billion monthly users, and Search was growing 10%. Now Search growth has accelerated to 17%.

The metrics Google celebrated on this call describe users staying on Google longer. Schindler described the new ad inventory as additive, reaching queries that were “previously challenging to monetize.”

That’s a monetization win for Google. The tradeoff to watch is referral traffic.

When asked about cannibalization, Pichai said Google hasn’t seen evidence of it:

“The combination of all of that I think creates an expansionary moment. I think it’s expanding the type of queries people do with Google overall.”

That may be true for queries. Whether it holds for referral traffic is something you’ll need to track in your own analytics.

Looking Ahead

Google maintains the position that AI features expand search activity rather than cannibalize it. The Q4 revenue numbers back it up.

The open question is what expanding AI Mode features means for referral traffic, and your own analytics will tell that story.


Featured Image: Rokas Tenys/Shutterstock

The Real SEO Skill No One Teaches: Problem Deduction via @sejournal, @billhunt

Most SEO failures are not optimization failures. They are reasoning failures that occur before optimization even begins.

In enterprise SEO escalations, the pattern is remarkably consistent. Teams jump straight to causes, debate theories, and assign blame before anyone clearly articulates the actual problem they are trying to understand.

Once blame enters the conversation, problem definition disappears. Teams shift into CYA mode, and without a shared understanding of the problem, every proposed fix becomes guesswork.

The Failure Pattern Everyone Recognizes

If you’ve worked in enterprise SEO long enough, you’ve seen this meeting.

A stakeholder raises an issue. Google is showing the wrong title or site name. Search visibility dropped. A location isn’t represented correctly. The room doesn’t go quiet. It fills with explanations.

Someone points to a lack of internal links. Another suggests Google rewrote the titles. Yet another CMS defect is mentioned. A recent Google update is blamed. Someone inevitably asks whether hreflang is broken.

Each explanation sounds plausible in isolation. Each reflects real experience. But none of them is grounded in a clearly stated problem.

Everyone is trying to be helpful. No one has actually said what outcome the system produced.

SEO discussions often collapse not because teams lack expertise, but because they skip the most important step: precisely describing the system outcome they are trying to explain.

Meeting Two: Activity Without Clarity

What usually follows is a second meeting. On the surface, it feels productive.

Teams arrive having done work. The CMS has been reviewed. A detailed technical SEO audit is complete. Google update trackers and industry forums have been checked for similar impacts, along with LinkedIn commentary. Multiple diagnostic tools have been run.

There is evidence of many man-hours of activity presented. There are screenshots of issues and non-issues, and it all looks like progress toward a resolution. In reality, it is often a misdirected effort.

If the original problem was vague or incorrectly framed, all of that analysis is aimed at the wrong target. Only later does the realization set in. While the audits detected issues, they are not related to this problem.

Time and attention were spent validating assumptions instead of diagnosing system behavior.

That’s not an execution failure. It’s a problem definition failure.

Why SEO Conversations Go Off The Rails

That failure isn’t accidental. It’s structural, and SEO is uniquely exposed to it.

I have often been critical, stating that the search industry lacks root cause analysis. That’s true, but it’s not because teams aren’t trying. There is no shortage of audits, checklists, or prescriptive processes when a traffic drop or SERP anomaly appears. The problem is that those tools narrow thinking rather than clarify it. They push teams toward doing something before anyone has agreed on what actually happened.

In many SEO conversations, signals are treated as probabilistic guesses rather than observed outcomes. Rankings fluctuate, a listing looks different, traffic dips, and the discussion quickly drifts toward familiar explanations. Google must have changed something. A ranking factor shifted. An update rolled out.

What gets missed is far more mundane and far more common. Control is spread across teams. Changes are made inside one department and are never communicated to another. Content, templates, navigation, schema, analytics, and infrastructure evolve independently. Cause and effect don’t move in straight lines, and no single team sees the whole system.

When no one clearly states the outcome the system produced, the group defaults to what feels responsible: activity.

Root cause analysis turns into a checklist exercise. Teams start debating causes before agreeing on the outcome itself. Meetings fill with effort, artifacts, and action items, but clarity never quite arrives.

Systems, however, don’t respond to effort. They respond to inputs.

The Missing Skill: Problem Deduction

The most important SEO skill isn’t keyword research, schema, technical audits, GEO, or any other optimization acronym that happens to be in fashion. Those are all processes and tools. Useful ones. But they only matter after the real work has been done. That work is problem deduction.

Problem deduction is the discipline of slowing the conversation down long enough to understand what the system actually produced, not what the team expected it to produce. It requires stepping outside of assumptions, resisting familiar explanations, and describing the outcome in neutral terms before trying to fix anything.

Only then does real analysis begin. Teams can reason backward through the signals that contributed to the outcome, distinguish between inputs they can change and constraints they inherited, and act without blame or superstition driving the discussion.

In practice, problem deduction means the ability to:

  • Observe a system outcome without bias, focusing on what the system produced rather than what was intended.
  • Describe that outcome precisely and neutrally, without embedding assumptions about cause.
  • Reason backward through contributing signals, identifying which inputs could plausibly influence the result.
  • Separate fixable inputs from historical constraints, so effort is spent where it can actually matter.
  • Act without blame or superstition, keeping decisions grounded in evidence rather than instinct.

This doesn’t replace technical SEO or root cause analysis. It makes them possible.

Problem deduction is systems thinking applied to search. And almost no one teaches it.

A Real-World Enterprise Example

Recently, I reviewed an enterprise case where a client was frustrated that Google consistently displayed a specific location as the site name, regardless of the user’s location or query intent. The conversation followed a familiar arc. At first, explanations came quickly. Someone pointed to internal linking, noting that this location had accumulated more authority over time. Others suggested Google’s automatic title rewrites were to blame. The CMS came up, along with the possibility of injected or inconsistent code. SEO implementation gaps were also mentioned. Each explanation sounded reasonable. All of them were based on real experience. But none of them described the outcome. So we stopped the discussion and reset the conversation by stating the problem plainly:

Google selected a location, not the brand name, as the site name representing the brand in search results.

That single sentence changed the tone of the room. Once the outcome was clearly defined, the reasoning became straightforward. The discussion shifted from speculation to diagnosis, and the signals that led to that result became much easier to trace.

How Google Actually Made That Decision

Google wasn’t confused. It was responding to a consistent set of reinforcing signals.

Once the outcome was clearly defined, the explanation stopped being mysterious. Several independent signals all pointed to the same conclusion, and Google simply followed the strongest, most consistent path.

1. Misapplied WebSite Schema

One issue started at the structural level. Location pages had been marked up as if each were a separate website entity, rather than reinforcing the primary brand domain. Multiple pages effectively claimed to be “the website,” diluting canonical authority and causing the schema signal to cancel itself out through duplication. Google didn’t misunderstand the markup. It received conflicting declarations and discounted them logically.

2. Title Tag Dilution

At the same time, title tags failed to reinforce a clear hierarchy. The homepage HTML title tag attempted to carry too much information at once, referencing the marketing tagline first, then the brand and first location, and finally the other locations, separated by commas, into a single tag. Instead of clarifying the relationship between the brand and locations, the structure blurred it. Google responded by favoring the location that was most consistently reinforced across signals. Google favored the most consistently reinforced location, not arbitrarily, but logically.

3. External Corroboration Bias

External signals reinforced the same outcome. Inbound links, citations, and references disproportionately pointed to a single location. From Google’s perspective, the broader web corroborated what on-site signals already suggested. One location appeared to represent the brand more clearly than the others. This wasn’t favoritism. It was corroboration.

What Could Be Easily Fixed And What Couldn’t

Once the actual problem was clearly identified, the conversation changed. The issue wasn’t that Google was behaving unpredictably. It was that something in the system was consistently telling Google to treat a single location as the site name rather than the brand itself.

With the problem framed that way, analysis became practical. Instead of debating theories, we could examine the systems that contributed to that outcome and begin correcting them. Just as importantly, it allowed us to distinguish between changes that could be made immediately and those that would require sustained effort.

Some corrections were straightforward. Because the schema was generated programmatically, the WebSite markup could be adjusted immediately to reinforce the primary brand entity. The brand team also agreed to simplify the homepage title, focusing it on the brand and tagline, while allowing individual location pages to carry the weight of location-specific signals.

Other signals were less malleable. External corroboration, built up through years of links and citations pointing to a single location, couldn’t be reversed quickly. That work would take time and consistent reinforcement.

Problem deduction didn’t just tell us what to fix. It told us where to start, what to expect, and how much effort each correction would realistically require.

SEO teams waste enormous effort trying to “fix” things that can only change gradually. Problem deduction helps teams focus on directional correction rather than instant reversal.

Why Root Cause Analysis Often Fails In SEO

Root cause analysis breaks down when teams try to answer why” before agreeing on “what.”

In enterprise SEO, that failure is amplified by how work is organized. Control is decentralized across content, engineering, analytics, brand, legal, localization, and platform teams. No single group owns the full system, yet everyone is accountable to their own KPIs. When an anomaly appears, the instinct isn’t to describe the outcome carefully. It’s to protect territory.

Conversations shift quickly. Causes are proposed before outcomes are defined. Responsibility is implied, then deflected. Each team points to the part of the system it doesn’t control. The discussion becomes less about understanding behavior and more about avoiding fault.

At the same time, the process itself narrows thinking. Root cause analysis turns into a checklist exercise. Teams reach for audits, tools, and familiar diagnostic steps, not because they are wrong, but because they are safe. Checklists create motion without requiring agreement, and activity becomes a substitute for clarity.

When internal explanations feel uncomfortable or politically risky, attention often shifts outward. Someone cites a recent Google update. Another references a post from a well-known SEO or a chart showing sector-wide volatility. External signals offer a kind of relief. If “everyone” is seeing impact, then no one internally has to explain their system.

But those signals are rarely diagnostic. Used too early, they short-circuit reasoning rather than support it.

The result is a familiar pattern. Meetings generate effort, artifacts, and action items, but the outcome itself remains vaguely defined. Teams stay busy. Nothing really changes.

Problem deduction interrupts that cycle. It forces agreement on what the system actually produced before explanations, defenses, or fixes enter the conversation. Once the outcome is clearly defined, decentralization becomes navigable, blame loses its power, and root cause analysis shifts from performance to purpose.

That’s when it starts working.

The Skill Enterprises Should Be Hiring For First

Not long ago, an advisory client asked me a deceptively simple question while defining a new enterprise search role.

“What is the single most important skill we should hire for?”

They were expecting a familiar answer. Something about technical SEO depth, AI search experience, schema expertise, or platform fluency. That’s usually how these conversations go.

I didn’t give them any of those. Instead, I said critical reasoning.

There was a pause.

Despite what many people in the search industry believe, technical skills are the easy part. Tools can be learned. Platforms change. Gaps get closed. Teams adapt. What’s far harder to teach is the ability to think clearly when the system doesn’t behave the way you expected it to.

Enterprise SEO is full of that kind of ambiguity. Signals conflict. Outcomes are indirect. Ownership is fragmented. And when things go wrong, pressure builds quickly.

In those moments, the people who struggle most aren’t the ones who lack tactical knowledge. They’re the ones who can’t slow the conversation down long enough to reason.

The skill that matters is the ability to observe what the system actually produced without bias, describe it precisely, separate symptoms from causes, reason backward through contributing signals, and resist the urge to jump to conclusions or assign blame.

In other words, problem deduction.

Specifically (as highlighted above), the ability to:

  • Observe a system outcome without bias.
  • Describe it precisely.
  • Separate symptoms from causes.
  • Reason backward through contributing signals.
  • Resist jumping to conclusions or assigning blame.

I told them plainly: We can teach the mechanics of search. What’s nearly impossible to teach is how to reason critically if that muscle isn’t already there. People either have it or they don’t. Enterprise SEO punishes the absence of that skill more than almost any other digital discipline.

This Is Bigger Than SEO

Once you recognize the pattern, it becomes hard to unsee.

The same failure mode that derails root cause analysis also explains why SEO so often turns political. When outcomes aren’t clearly defined, teams fill the gap with narratives. Best practices harden into superstition. Google updates become a convenient external explanation for internal incoherence. Infrastructure issues quietly masquerade as ranking problems because they’re harder to confront directly.

None of this happens because teams are careless. It happens because modern digital systems are fragmented by design.

As described earlier, control is decentralized across content, engineering, analytics, brand, legal, localization, and platform teams. No one owns the entire system, yet everyone is accountable to their own KPIs. When something goes wrong, describing the outcome precisely feels risky. It invites scrutiny. It raises uncomfortable questions about ownership and handoffs.

So conversations drift. Causes are debated before outcomes are agreed upon. Responsibility is implied, then deflected. Checklists replace reasoning because they allow motion without alignment. And when internal explanations feel politically unsafe, attention shifts outward – to Google updates, industry chatter, or gurus diagnosing sector-wide volatility.

Those external signals provide relief, but not resolution. They describe correlation, not causation. They offer context, not clarity and allow organizations to stay busy without ever confronting how their own systems produced the result.

This is where SEO begins to overlap with something broader: findability.

Whether someone encounters a brand through Google, an AI assistant, a marketplace, or a vertical search engine, the underlying questions are the same. Are we present? Are we represented clearly and consistently? Does that representation invite deeper engagement, or does it confuse and fragment trust?

Those outcomes don’t depend on isolated optimizations. They depend on coherent systems that behave predictably across surfaces.

Problem deduction is what makes that coherence possible. By forcing agreement on what the system actually produced before explanations or fixes enter the room, it cuts through decentralization, neutralizes blame, and restores reasoning. Root cause analysis stops being performative and starts serving its purpose.

That’s when the conversation changes. And that’s when progress actually begins.

The Real Takeaway

Google didn’t choose the wrong site name. It chose the only version of the brand the system clearly defined.

The real SEO skill isn’t knowing what to change. It’s knowing what actually happened before you touch anything at all.

Until enterprises teach, hire for, and reward problem deduction, SEO conversations will continue to spin in circles, fixing symptoms while the system quietly reinforces the same outcomes.

And no amount of optimization can fix a problem that was never clearly defined in the first place.

More Resources:


Featured Image: KitohodkA/Shutterstock

Information Retrieval Part 2: How To Get Into Model Training Data

There has never been a more important time in your career to spend time learning and understanding. Not because AI search differs drastically from traditional search. But because everyone else thinks it does.

Every C-suite in the country is desperate to get this right. Decision-makers need to feel confident that you and I are the right people to lead us into the new frontier.

We need to learn the fundamentals of information retrieval. Even if your business shouldn’t be doing anything differently.

Here, that starts with understanding the basics of model training data. What is it, how does it work and – crucially – how do I get in it.

TL;DR

  1. AI is the product of its training data. The quality (and quantity) the model trains on is key to its success.
  2. The web-sourced AI data commons is rapidly becoming more restricted. This will skew data representativity, freshness, and scaling laws.
  3. The more consistent, accurate brand mentions you have that appear in training data, the less ambiguous you are.
  4. Quality SEO, with better product and traditional marketing, will improve your appearance in the training and data, and eventually with real-time RAG/retrieval.

What Is Training Data?

Training data is the foundational dataset used in training LLMs to predict the most appropriate next word, sentence, and answer. The data can be labeled, where models are taught the right answer, or unlabeled, where they have to figure it out for themselves.

Without high-quality training data, models are completely useless.

From semi-libelous tweets to videos of cats and great works of art and literature that stand the test of time, nothing is off limits. Nothing. It’s not just words either. Speech-to-text models need to be trained to respond to different speech patterns and accents. Emotions even.

Image Credit: Harry Clarkson-Bennett

How Does It Work?

The models don’t memorize, they compress. LLMs process billions of data points, adjusting internal weights through a mechanism known as backpropagation.

If the next word predicted in a string of training examples is correct, it moves on. If not, it gets the machine equivalent of Pavlovian conditioning.

Bopped on the head with a stick or a “good boy.”

The model is then able to vectorize. Creating a map of associations by term, phrase, and sentence.

  • Converting text into numerical vectors, aka Bag of Words.
  • Capturing semantic meaning of words and sentences, preserving wider context and meaning (word and sentence embeddings).

Rules and nuances are encoded as a set of semantic relationships; this is known as parametric memory. “Knowledge” baked directly into the architecture. The more refined a model’s knowledge on a topic, the less it has to use a form of grounding to verify its twaddle.

Worth noting that models with a high parametric memory are faster at retrieving accurate information (if available), but have a static knowledge base and literally forget things.

RAG and live web search is an example of a model using non-parametric memory. Infinite scale, but slower. Much better for news and when results require grounding.

Crafting Better Quality Algorithms

When it comes to the training data, drafting better quality algorithms relies on three elements:

  1. Quality.
  2. Quantity.
  3. Removal of bias.

Quality of data matters for obvious reasons. If you train a model on poorly labeled, solely synthetic data, the model performance cannot be expected to exactly mirror real problems or complexities.

Quantity of data is a problem, too. Mainly because these companies have eaten everything in sight and done a runner on the bill.

Leveraging synthetic data to solve issues of scale isn’t necessarily a problem. The days of accessing high-quality, free-to-air content on the internet for these guys are largely gone. For two main reasons:

  1. Unless you want diabolical racism, mean comments, conspiracy theories, and plagiarized BS, I’m not sure the internet is your guy anymore.
  2. If they respect company’s robots.txt directives at least. Eight in 10 of the world’s biggest news websites now block AI training bots. I don’t know how effective their CDN-level blocking is, but this makes quality training data harder to come by.

Bias and diversity (or lack of it) is a huge problem too. People have their own inherent biases. Even the ones building these models.

Shocking I know…

If models are fed data unfairly weighted towards certain characteristics or brands, it can reinforce societal issues. It can further discrimination.

Remember, LLMs are neither intelligent nor databases of facts. They analyze patterns from ingested data. Billions or trillions of numerical weights that determine the next word (token) following another in any given context.

How Is Training Data Collected?

Like every good SEO, it depends.

  1. If you built an AI model explicitly to identify pictures of dogs, you need pictures of dogs in every conceivable position. Every type of dog. Every emotion the pooch shows. You need to create or procure a dataset of millions, maybe billions, of canine images.
  2. Then it must be cleaned. Think of it as structuring data into a consistent format. In said dog scenario, maybe a feline friend nefariously added pictures of cats dressed up as dogs to mess you around. Those must be identified.
  3. Then labeled (for supervised learning). Data labeling (with some human annotation) ensures we have a sentient being somewhere in the loop. Hopefully, an expert to add relevant labels to a tiny portion data, so that a model can learn. For example, a dachshund sitting on a box looking melancholic.
  4. Pre-processing. Responding to issues like cats masquerading as dogs. Ensuring you minimize potential biases in the dataset like specific dog breeds being mentioned far more frequently than others.
  5. Partitioned. A portion of the data is kept back so the model can’t memorise the outputs. This is the final validation stage. Kind of like a placebo.

This is, obviously, expensive and time-consuming. It’s not feasible to take up hundreds of thousands of hours of expertise from real people in fields that matter.

Think of this. You’ve just broken your arm, and you’re waiting in the ER for six hours. You finally get seen, only to be told you had to wait because all the doctors have been processing images for OpenAI’s new model.

“Yes sir, I know you’re in excruciating pain, but I’ve got a hell of a lot of sad looking dogs to label.”

Data labeling is a time-consuming and tedious process. To combat this, many businesses hire large teams of human data annotators (aka humans in the loop, you know, actual experts), assisted by automated weak labeling models. In supervised learning, they sort the initial labeling.

For perspective, one hour of video data can take humans up to 800 hours to annotate.

Micro Models

So, companies build micro-models. Models that don’t require as much training or data to run. The humans in the loop (I’m sure they have names) can start training micro-models after annotating a few examples.

The models learn. They train themselves.

So over time, human input decreases, and we’re only needed to validate the outputs. And to make sure the models aren’t trying to undress children, celebrities, and your coworkers on the internet.

But who cares about that in the face of “progress.”

Image Credit: Harry Clarkson-Bennett

Types Of Training Data

Training data is usually categorized by how much guidance is provided or required (supervision) and the role it plays in the model’s lifecycle (function).

Ideally a model is largely trained on real data.

Once a model is ready, it can be trained and fine-tuned on synthetic data. But synthetic data alone is unlikely to create high-quality models.

  • Supervised (or labeled): Where every input is annotated with the “right” answer.
  • Unsupervised (or unlabeled): Work it out yourself, robots, I’m off for a beer.
  • Semi-supervised: where a small amount of the data is properly labeled and model “understands” the rules. More, I’ll have a beer in the office.
  • RLHF (Reinforcement Learning from Human Feedback): humans are shown two options and asked to pick the “right” one (preference data). Or a person demonstrates the task at hand for the mode to imitate (demonstration data).
  • Pre-training and fine-tuning data: Massive datasets allow for broad information acquisition, and fine-tuning is used to turn the model into a category expert.
  • Multi-modal: Images, videos, text, etc.

Then some what’s known as edge case data. Data designed to “trick” the model to make it more robust.

In light of the let’s call it “burgeoning” market for AI training data, there are obvious issues of “fair use” surrounding it.

“We find that 23% of supervised training datasets are published under research or non-commercial licenses.”

So pay people.

The Spectrum Of Supervision

In supervised learning, the AI algorithm is given labeled data. These labels define the outputs and are fundamental to the algorithm being able to improve over time on its own.

Let’s say you’re training a model to identify colors. There are dozens of shades of each color. Hundreds even. So while this is an easy example, it requires accurate labeling. The problem with accurate labeling is its time-consuming and potentially costly.

In unsupervised learning, the AI model is given unlabeled data. You chuck millions of rows, images, or videos at a machine, sit down for a coffee, and then kick it when it hasn’t worked out what to do.

It allows for more exploratory “pattern recognition.” Not learning.

While this approach has obvious drawbacks, it’s incredibly useful at identifying patterns a human might miss. The model can essentially define its own labels and pathway.

Models can and do train themselves, and they will find things a human never could. They’ll also miss things. It’s like a driverless car. Driverless cars may have fewer accidents than when a human is in the loop. But when they do, we find it far more unpalatable.

We don’t trust tech autonomy. (Image Credit: Harry Clarkson-Bennett)

It’s the technology that scares us. And rightly so.

Combatting Bias

Bias in training data is very real and potentially very damaging. There are three phases:

  1. Origin bias.
  2. Development bias.
  3. Deployment bias.

Origin bias references the validity and fairness of the dataset. Is the data all-encompassing? Is there any obvious systemic, implicit, or confirmation bias present?

Development bias includes the features or tenets of the data the model is being trained on. Does algorithmic bias occur because of the training data?

Then we have deployment bias. Where the evaluation and processing of the data leads to flawed outputs and automated/feedback loop bias.

You can really see why we need a human in the loop. And why AI models training on synthetic or inappropriately chosen data would be a disaster.

In healthcare, data collection activities influenced by human bias can lead to the training of algorithms that replicate historical inequalities. Yikes.

Leading to a pretty bleak cycle of reinforcement.

The Most Frequently Used Training Data Sources

Training data sources are wide-ranging in both quality and structure. You’ve got the open web, which is obviously a bit mental. X, if you want to train something to be racist. Reddit, if you’re looking for the Incel Bot 5000.

Or highly structured academic and literary repositories if you want to build something, you know, good … Obviously then you have to pay something.

Common Crawl

Common Crawl is a public web repository, a free, open-source storehouse of historical and current web crawl data available to pretty much anyone on the internet.

The full Common Crawl Web Graph currently contains around 607 million domain records across all datasets, with each monthly release covering 94 to 163 million domains.

In the Mozilla Foundation’s 2024 report, Training Data for the Price of a Sandwich, 64% of the 47 LLMs analysed used at least one filtered version of Common Crawl data.

If you aren’t in the training data, you’re very unlikely to be cited and referenced. The Common Crawl Index Server lets you search any URL pattern against their crawl archives and Metehan’s Web Graph helps you see how “centered you are.”

Wikipedia (And Wikidata)

The default English Wikipedia dataset contains 19.88 GB of complete articles that help with language modeling tasks. And Wikidata is an enormous, incredibly comprehensive knowledge graph. Immensely structured data.

While representing only a small percentage of the total tokens, Wikipedia is perhaps the most influential source for entity resolution and factual consensus. It is one of the most factually accurate, up-to-date, and well-structured repositories of content in existence.

Some of the biggest guys have just signed deals with Wikipedia.

Publishers

OpenAI, Gemini, etc., have multi-million dollar licensing deals with a number of publishers.

The list goes on, but only for a bit … and not recently. I’ve heard things have clammed shut. Which, given the state of their finances, may not be surprising.

Media & Libraries

This is mainly for multi-modal content training. Shutterstock (images/video), Getty Images have one with Perplexity, and Disney (a 2026 partner for the Sora video platform) provides the visual grounding for multi-modal models.

As part of this three-year licensing agreement with Disney, Sora will be able to generate short, user-prompted social videos based on Disney characters.

As part of the agreement, Disney will make a $1 billion equity investment in OpenAI, and receive warrants to purchase additional equity.

Books

BookCorpus turned scraped data of 11,000 unpublished books into a 985 million-word dataset.

We cannot write books fast enough for models to continually learn on. It’s part of the soon to happen model collapse.

Code Repositories

Coding has become one of the most influential and valuable features of LLMs. Specific LLMs like Cursor or Claude Code are incredible. GitHub and Stack Overflow data have built these models.

They’ve built the vibe-engineering revolution.

Public Web Data

Diverse (but relevant) web data results in faster convergence during training, which in turn reduces computational requirements. It’s dynamic. Ever-changing. But, unfortunately, a bit nuts and messy.

But, if you need vast swathes of data, maybe in real-time, then public web data is the way forward. Ditto for real opinions and reviews of products and services. Public web data, review platforms, UGC, and social media sites are great.

Why Models Aren’t Getting (Much) Better

While there’s no shortage of data in the world, most of it is unlabeled and, thus, can’t actually be used in supervised machine learning models. Every incorrect label has a negative impact on a model’s performance.

According to most, we’re only a few years away from running out of quality data. Inevitably, this will lead to a time when those genAI tools start consuming their own garbage.

This is a known problem that will cause model collapse.

  • They are being blocked by companies that do not want their data used pro bono to train the models.
  • Robots.txt protocols (a directive, not something directly enforceable), CDN-level blocking, and terms of service pages have been updated to tell these guys to get lost.
  • They consume data quicker than we can produce it.

Frankly, as more publishers and websites are forced into paywalling (a smart business decision), the quality of these models only gets worse.

So, How Do You Get In The Training Data?

There are two obvious approaches I think of.

  1. To identify the seed data sets of models that matter and find ways into them.
  2. To forgo the specifics and just do great SEO and wider marketing. Make a tangible impact in your industry.

I can see pros and cons to both. Finding ways into specific models is probably highly unnecessary for most brands. To me this smells more like grey hat SEO. Most brands will be better off just doing some really good marketing and getting shared, cited and you know, talked about.

These models are not trained on directly up-to-date data. This is important because you cannot retroactively get into a specific model’s training data. You have to plan ahead.

If you’re an individual, you should be:

  • Creating and sharing content.
  • Going on podcasts.
  • Attending industry events.
  • Sharing other people’s content.
  • Doing webinars.
  • Getting yourself in front of relevant publishers, publications, and people.

There are some pretty obvious sources of highly structured data that models have paid for in recent times. I know, they’ve actually paid for it. I don’t know what the guys at Reddit and Wikipedia had to do to get money from these guys, and maybe I don’t want to.

How Can I Tell What Datasets Models Use?

Everyone has become a lot more closed off with what they do and don’t use for training data. I suspect this is both legally and financially motivated. So, you’ll need to do some digging.

And there are some massive “open source” datasets I suspect they all use:

  • Common Crawl.
  • Wikipedia.
  • Wikidata.
  • Coding repositories.

Fortunately, most deals are public, and it’s safe to assume that models use data from these platforms.

Google has a partnership with Reddit and access to an insane amount of transcripts from YouTube. They almost certainly have more valuable, well-structured data at their fingertips than any other company.

Grok trained almost exclusively on real-time data from X. Hence why it acts like a pre-pubescent school shooter and undresses everyone.

Worth noting that AI companies use third party vendors. Factories where data is scraped, cleaned and structured to create supervised datasets. Scale AI is the data engine that the big players use. Bright Data specialise in web data collection.

A Checklist

OK, so we’re trying to feature in parametric memory. To appear in the LLMs training data so the model recognizes you and you’re more likely to be used for RAG/retrieval. That means we need to:

  1. Manage the multi-bot ecosystem of training, indexing, and browsing.
  2. Entity optimization. Well-structured, well-connected content, consistent NAPs, sameAs schema properties, and Knowledge Graph presence. In Google and Wikidata.
  3. Make sure your content is rendered on the server side. Google has become very adept at rendering content on the client side. Bots like GPT-bot only see the HTML response. JavaScript is still clunky.
  4. Well-structured, machine-readable content in relevant formats. Tables, lists, properly structured semantic HTML.
  5. Get. Yourself. Out. There. Share your stuff. Make noise.
  6. Be ultra, ultra clear on your website about who you are. Answer the relevant questions. Own your entities.

You have to balance direct associations (what you say) with semantic associations (what others say about you). Make your brand the obvious next word.

Modern SEO, with better marketing.

More Resources:


Read Leadership In SEO, subscribe now.


Featured Image: Collagery/Shutterstock

Using AI For SEO Can Fail Without Real Data (& How Ahrefs Fixes It) via @sejournal, @ahrefs

This post was sponsored by Ahrefs. The opinions expressed in this article are the sponsor’s own.

If you’ve ever run into the limits of solo AI or manual SEO tools, this article is for you.

AI on its own can write and suggest ideas, but without reliable data to anchor those suggestions, it can miss the mark. On the other hand, traditional SEO dashboards are powerful – yet slow and siloed. The emerging sweet spot? Connecting AI to real, live SEO data so you can ask natural language questions and get deep answers fast.

Ahrefs Uses Its Own MCP Server & It Improves SEO Workflows

At its core, MCP stands for Model Context Protocol – an open standard that lets compatible AI assistants (like ChatGPT and Claude) directly access external data sources and tools through a standardized connection. This means you can ask your AI assistant questions like “which keywords my competitor ranks for that I don’t” or “which sites are gaining the most organic traffic this year” – and get answers based on real, up-to-date SEO data instead of guesses.

Imagine you’re planning to launch a new eCommerce product. Instead of manually exporting CSVs from multiple dashboards and painstakingly combining them, you could simply prompt an AI assistant to pull competitive insights, keyword opportunities, and content ideas directly from a connected SEO dataset – all in one place. That’s the power of an MCP integration.

Why AI + Real SEO Data Together Beats Guessing Or Generic Prompts

Most marketers use at least two types of tools: dedicated SEO platforms (for data) and AI assistants (for speed and interpretation). However:

  • AI on its own can hallucinate – it generates plausible-sounding answers, but without live data, those answers may be inaccurate or outdated.
  • SEO dashboards by themselves are often slow – you click around multiple screens, export reports, and manually interpret results.
  • Humans still need to make strategic decisions – but data plus AI frees up your time to focus on strategy, not grunt work.

Connecting AI to a live SEO dataset unites the best of both worlds: the intelligence and language fluency of modern AI with the accuracy and scale of professional SEO metrics.

15 Practical Use Cases & Prompts To Ask Your SEO AI Agent

Below are real prompt ideas and workflows you can incorporate into your planning, competitive research, and SEO execution. These are grouped from simple (fast answers) to advanced (deep analysis) – and all are grounded in actionable insights you can use today.

Level 1: Quick Insights You Can Get in Minutes

These are great for rapid decision-making and daily checks.

1. Identify Sites Growing Organic Traffic

Ask your AI:

Which of these 10 competitors has grown organic search traffic the most over the last 12 months?
This lets you quickly spot who is gaining momentum – and why – without manual reporting.

2. Find Competitor Rankings You Don’t Rank For

Tell me which first-page Google rankings [Competitor A] has that [My Site] doesn’t.
This gives you a direct gap list you can use for content or optimization ideas.

3. Most Linked-To Pages on Any Domain

List the top 10 pages on [domain] by number of backlinks, and show their estimated traffic.
This helps you spot proven content winners and consider similar formats.

4. Identify Organic Competitors

Give me a list of the closest organic search competitors for [My Site].
Great for broadening your competitive set beyond the obvious brands.

5. Combine Keyword Research With Headline Ideas

Help me find keywords people use before buying [product], and suggest related blog post headlines.
This blends keyword discovery with content planning in one step.

Level 2: Intermediate, More Strategic Queries

These involve deeper insights and slightly longer processing time.

6. Find Trending Keywords (and Why)

Show up to 20 trending keywords in my niche that may grow in popularity next year – include explanations.
This is better than a static list – you get context and rationale.

7. Analyze Multiple Domains at Scale

Give me a table of these 20 domains with Domain Rating, Organic Traffic, and number of top-3 rankings.
Great for benchmarking and competitor comparison.

8. Structure an Article With Keyword Insights

Help me build an article outline for [topic] based on keyword research.
This combines research with SEO content planning.

9. Top Ranking Sites for Specific Keyword Set

Among these keyphrases, tell me which sites rank in the highest positions.
Very helpful when exploring emerging niches within broader topics.

10. Find Broken Backlinks for Outreach Opportunities

Identify broken backlinks in this subfolder with high-authority referring domains.
Perfect for targeted link building.

Level 3: Advanced, High-Impact Research

These take more data and processing – but return strategic intelligence you can act on.

11. International SEO Expansion Ideas

Find similar businesses that have expanded into new countries and show where their organic traffic is growing.
A great way to spot untapped markets.

12. Competitor Content Strategy Deep Dive

Analyze top organic competitors and show their content themes, unique angles, and ranking patterns.
This helps refine your content planning with context beyond just keywords.

13. Comprehensive Site SEO Recommendations

You are an SEO expert with access to extensive data – offer recommendations to grow organic traffic for [brand].
This leverages the AI to synthesize data into strategic advice you can execute.

14. In-Depth Industry Ranking Patterns

Provide a list of top keyphrases where a site ranks first-page and includes certain SERP features.
Used for deep pattern discovery in competitive environments.

15. Multi-Domain Backlink Profile Analysis

Show backlink acquisition rates for these five competitors.
Useful for assessing link velocity and authority-building trends.

Tips to Get More Out of Data-Driven AI Prompts

Use these best practices to ensure your AI assistant actually retrieves the correct data:

  • Always specify that you want results from the SEO dataset rather than web search.
  • Include clear context (e.g., competitors, timeframes, regions).
  • Be explicit about limits (e.g., “show only keyword opportunities with volume > X”).
  • Track your usage and data limits via your SEO dashboard so you don’t hit quotas unexpectedly.

Image Credits

Featured Image: Image by Ahrefs. Used with permission.

GSC Data Is 75% Incomplete via @sejournal, @Kevin_Indig

Boost your skills with Growth Memo’s weekly expert insights. Subscribe for free!

My findings this week show Google Search Console data is about 75% incomplete, making single-source GSC decisions dangerously unreliable.

Google filters 3/4 of search impressions for “privacy,” while bot inflation and AIOs corrupt what remains. (Image Credit: Kevin Indig)

1. GSC Used To Be Ground Truth

Search Console data used to be the most accurate representation of what happens in the search results. But privacy sampling, bot-inflated impressions, and AI Overview (AIO) distortion suck the reliability out of the data.

Without understanding how your data is filtered and skewed, you risk drawing the wrong conclusions from GSC data.

SEO data has been on a long path of becoming less reliable, starting with Google killing keyword referrer to excluding critical SERP Features from performance results. But three key events over the last 12 months topped it off:

  • January 2025: Google deploys “SearchGuard,” requiring JavaScript and (sophisticated) CAPTCHA for anyone looking at search results (turns out, Google uses a lot of advanced signals to differentiate humans from scrapers).
  • March 2025: Google significantly amps up the number of AI Overviews in the SERPs. We’re seeing a significant spike in impressions and drop in clicks.
  • September 2025: Google removes num=100 parameter, which SERP scrapers use to parse the search results. The impression spike normalizes, clicks stay down.

On one hand, Google took measures to clean up GSC data. On the other hand, the data still leaves us with more open questions than answers.

2. Privacy Sampling Hides 75% Of Queries

Google filters out a significant amount of impressions (and clicks) for “privacy” reasons. One year ago, Patrick Stox analyzed a large dataset and came to the conclusion that almost 50% are filtered out.

I repeated the analysis (10 sites in B2B out of the USA) across ~4 million clicks and ~450 million impressions.

Methodology:

  • Google Search Console (GSC) provides data through two API endpoints that reveal its filtering behavior. The aggregate query (no dimensions) returns total clicks and impressions, including all data. The query-level query (with “query” dimension) returns only queries meeting Google’s privacy threshold.
  • By comparing these two numbers, you can calculate the filter rate.
  • For example, if aggregate data shows 4,205 clicks but query-level data only shows 1,937 visible clicks, Google filtered 2,268 clicks (53.94%).
  • I analyzed 10 B2B SaaS sites (~4 million clicks, ~450 million impressions), comparing 30-day, 90-day, and 12-month periods against the same analysis from 12 months prior.

My conclusion:

1. Google filters out ~75% of impressions.

Image Credit: Kevin Indig
  • The filter rate on impressions is incredibly high, with three-fourths filtered for privacy.
  • 12 months ago, the rate was only 2 percentage points higher.
  • The range I observed went from 59.3% all the way up to 93.6%.
Image Credit: Kevin Indig

2. Google filters out ~38% of clicks, but ~5% less than 12 months ago.

IMage Credit: Kevin Indig
  • Click filtering is not something we talk about a lot, but it seems Google doesn’t report up to one-third of all clicks that happened.
  • 12 months ago, Google filtered out over 40% of clicks.
  • The range of filtering spans from 6.7% to 88.5%!
Image Credit: Kevin Indig

The good news is that the filter rate has gone slightly down over the last 12 months, probably as a result of fewer “bot impressions.”

The bad news: The core problem persists. Even with these improvements, 38% click-filtering and 75% impression-filtering remain catastrophically high. A 5% improvement doesn’t make single-source GSC decisions reliable when three-fourths of your impression data is missing.

3. 2025 Impressions Are Highly Inflated

Image Credit: Kevin Indig

The last 12 months show a rollercoaster of GSC data:

  • In March 2025, Google intensified the rollout of AIOs and showed 58% more for the sites I analyzed.
  • In July, impressions grew by 25.3% and by another 54.6% in August. SERP scrapers somehow found a way around SearchGuard (the protection “bot” that Google uses to prevent SERP scrapers) and caused “bot impressions” to capture AIOs.
  • In September, Google removed the num=100 parameter, which caused impressions to drop by 30.6%.
Image Credit: Kevin Indig

Fast forward to today:

  • Clicks decreased by 56.6% since March 2025.
  • Impressions normalized (down -9.2%).
  • AIOs reduced by 31.3%.

I cannot come to a causative number of reduced clicks from AIOs, but the correlation is strong: 0.608. We know AIOs reduce clicks (makes logical sense), but we don’t know exactly how much. To figure that out, I’d have to measure CTR for queries before and after an AIO shows up.

But how do you know click decline is due to an AIO and not just poor content quality or content decay?

Look for temporal correlation:

  • Track when your clicks dropped against Google’s AIO rollout timeline (March 2025 spike). Poor content quality shows gradual decline; AIO impact is sharp and query-specific.
  • Cross-reference with position data. If rankings hold steady while clicks drop, that signals AIO cannibalization. Check if the affected queries are informational (AIO-prone) vs. transactional (AIO-resistant). Your 0.608 correlation coefficient between AIO presence and click reduction supports this diagnostic approach.

4. Bot Impressions Are Rising

Image Credit: Kevin Indig

I have reason to believe that SERP scrapers are coming back. We can measure the amount of impressions likely caused by bots by filtering out GSC data by queries that contain more than 10 words and two impressions. The chance that such a long query (prompt) is used by a human twice is close to zero.

The logic of bot impressions:

  • Hypothesis: Humans rarely search for the exact same 5+ word query twice in a short window.
  • Filter: Identify queries with 10+ words that have >1 impression but zero clicks.
  • Caveat: This method may capture some legitimate zero-click queries, but provides a directional estimate of bot activity.

I compared those queries over the last 30, 90, and 180 days:

  • Queries with +10 words and +1 impression grew by 25% over the last 180 days.
  • The range of bot impressions spans from 0.2% to 6.5% (last 30 days).

Here’s what you can anticipate as a “normal” percentage of bot impressions for a typical SaaS site:

  • Based on the 10-site B2B dataset, bot impressions range from 0.2% to 6.5% over 30 days, with queries containing 10+ words and 2+ impressions but 0 clicks.
  • For SaaS specifically, expect a 1-3% baseline for bot impressions. Sites with extensive documentation, technical guides, or programmatic SEO pages trend higher (4-6%).
  • The 25% growth over 180 days suggests scrapers are adapting post-SearchGuard. Monitor your percentile position within this range more than the absolute number.

Bot impressions do not affect your actual rankings – just your reporting by inflating impression counts. The practical impact? Misallocated resources if you optimize for inflated impression queries that humans never search for.

5. The Measurement Layer Is Broken

Single-source decisions based on GSC data alone become dangerous:

  • Three-fourths of impressions are filtered.
  • Bot impressions generate up to 6.5% of data.
  • AIOs reduce clicks by over 50%.
  • User behavior is structurally changing.

Your opportunity is in the methodology: Teams that build robust measurement frameworks (sampling rate scripts, bot-share calculations, multi-source triangulation) have a competitive advantage.


Featured Image: Paulo Bobita/Search Engine Journal

Why SEO Roadmaps Break In January (And How To Build Ones That Survive The Year) via @sejournal, @cshel

SEO roadmaps have a lot in common with New Year’s resolutions: They’re created with optimism, backed by sincere intent, and abandoned far sooner than anyone wants to admit.

The difference is that most people at least make it to Valentine’s Day before quietly deciding that daily workouts or dry January were an ambitious, yet misguided, experiment. SEO roadmaps often start unraveling while Punxsutawney Phil is still deep in REM sleep.

By the third or fourth week of the year, teams are already making “temporary” adjustments. A content cadence slips here. A technical initiative gets deprioritized there. A dependency turns out to be more complicated than anticipated, etc. None of this is framed as failure, naturally, but the original plan is already being renegotiated.

This doesn’t happen because SEO teams are bad at planning. It happens because annual SEO roadmaps are still built as if search were a stable environment with predictable inputs and outcomes.

(Narrator: Search is not, and has never been, a stable environment with predictable inputs or outcomes.)

In January, just like that diet plan, the SEO roadmap looks entirely doable. By February, you’re hiding in a dark pantry with a sleeve of Thin Mints, and the roadmap is already in tatters.

Here’s why those plans break so quickly and how to replace them with a planning model that holds up once the year actually starts moving.

The January Planning Trap

Annual SEO roadmaps are appealing because they feel responsible.

  • They give leadership something concrete to approve.
  • They make resourcing look predictable.
  • They suggest that search performance can be engineered in advance.

Except SEO doesn’t operate in a static system, and most roadmaps quietly assume that it does.

By the time Q1 is halfway over, teams are already reacting instead of executing. The plan didn’t fail because it was poorly constructed. It failed because it was built on outdated assumptions about how search works now.

Three Assumptions That Break By February

1. Algorithms Behave Predictably Over A 12-Month Period

Most annual roadmaps assume that major algorithm shifts are rare, isolated events.

That’s no longer true.

Search systems are now updated continuously. Ranking behavior, SERP layouts, AI integrations, and retrieval logic evolve incrementally –  often without a single, named “update” to react to.

A roadmap that assumes stability for even one full quarter is already fragile.

If your plan depends on a fixed set of ranking conditions remaining intact until December, it’s already obsolete.

2. Technical Debt Stays Static Unless Something “Breaks”

January plans usually account for new technical work like migrations, performance improvements, structured data, internal linking projects.

What they don’t account for is technical debt accumulation.

Every CMS update, plugin change, template tweak, tracking script, and marketing experiment adds friction. Even well-maintained sites slowly degrade over time.

Most SEO roadmaps treat technical SEO as a project with an end date. In reality, it’s a system that requires continuous maintenance.

By February, that invisible debt starts to surface – crawl inefficiencies, index bloat, rendering issues, or performance regressions – none of which were in the original plan.

3. Content Velocity Produces Linear Returns

Many annual SEO plans assume that content output scales predictably:

More content = more rankings = more traffic

That relationship hasn’t been linear for a long time.

Content saturation, intent overlap, internal competition, and AI-driven summaries all flatten returns. Publishing at the same pace doesn’t guarantee the same impact quarter over quarter.

By February, teams are already seeing diminishing returns from “planned” content and scrambling to justify why performance isn’t tracking to projections.

What Modern SEO Roadmap Planning Actually Looks Like

Roadmaps don’t need to disappear, but they do need to change shape.

Instead of a rigid annual plan, resilient SEO teams operate on a quarterly diagnostic model, one that assumes volatility and builds flexibility into execution.

The goal isn’t to abandon strategy. It’s to stop pretending that January can predict December.

A resilient model includes:

  • Quarterly diagnostic checkpoints, not just quarterly goals.
  • Rolling prioritization, based on what’s actually happening in search.
  • Protected capacity for unplanned technical or algorithmic responses.
  • Outcome-based planning, not task-based planning.

This shifts SEO from “deliverables by date” to “decisions based on signals.”

The Quarterly Diagnostic Framework

Instead of locking a yearlong roadmap, break planning into repeatable quarterly cycles:

Step 1: Assess (What Changed?)

At the start of each quarter, and ideally again mid-quarter, evaluate:

  • Crawl and indexation patterns.
  • Ranking volatility across key templates.
  • Performance deltas by intent, not just keywords.
  • Content cannibalization and decay.
  • Technical regressions or new constraints.

This is not a full audit. It’s a focused diagnostic designed to surface friction early.

Step 2: Diagnose (Why Did It Change?)

This is where most roadmaps fall apart: They track metrics but skip interpretation.

Diagnosis means asking:

  • Is this decline structural, algorithmic, or competitive?
  • Did we introduce friction, or did the ecosystem change around us?
  • Are we seeing demand shifts or retrieval shifts?

Without this layer, teams chase symptoms instead of causes.

Step 3: Fix (What Actually Matters Now?)

Only after diagnosis should priorities shift. That shift may involve pausing content production, redirecting engineering resources, or deliberately doing nothing while volatility settles. Resilient planning accepts that the “right” work in February may bear little resemblance to what was approved in January.

How To Audit Mid-Quarter Without Panicking

Mid-quarter reviews don’t mean throwing out the plan. They mean stress-testing it.

A healthy mid-quarter SEO check should answer three questions:

  1. What assumptions no longer hold?
  2. What work is no longer high-leverage?
  3. What risk is emerging that wasn’t visible before?

If the answer to any of those changes execution, that’s not failure. It’s adaptive planning.

The teams that struggle are the ones afraid to admit the plan needs to change.

The Bottom Line

The acceleration introduced by AI-driven retrieval has shortened the gap between planning and obsolescence.

January SEO roadmaps don’t fail because teams lack strategy. They fail because they assume a level of stability that search has not offered in years. If your SEO plan can’t absorb algorithmic shifts, technical debt, and nonlinear content returns, it won’t survive the year. The difference between teams that struggle and teams that adapt is simple: One plans for certainty, the other plans for reality.

The teams that win in search aren’t the ones with the most detailed January roadmap. They’re the ones that can still make good decisions in February.

More Resources:


Featured Image: Anton Vierietin/Shutterstock

Better Metrics for AI Search Visibility

The rise of AI-generated search and discovery is pushing merchants to measure their products’ visibility on those platforms. Many search optimizers are attempting to apply traditional metrics such as traffic from genAI and rankings in the answers. Both fall short.

Traffic. Focusing on traffic obscures the purpose of AI answers: to satisfy a need on-site, not to generate clicks.

AI-generated solutions do not typically include links to branded websites. Google’s AI Overviews, for example, sometimes links product names to organic search listings.

Thus visibility does not equate to traffic. A merchant’s products could appear in an AI answer and receive no clicks.

Screenshot of the AI Overview showing the North Face citation and link to an organic listing.

Brand names cited in Google’s AI Overviews often link to organic search listings, such as this example for North Face hiking boots.

Rankings. AI answers often include lists. Many sellers are trying to track those lists to rank at or near the top. Yet tracking such rankings is impossible.

AI answers are unpredictable. A recent study by Sparktoro found that AI platforms recommend different brands and different orders every time the same person asks the same question.

Better AI Metrics

Here are better metrics to measure AI visibility.

Product or brand positioning in LLM training data

Training data is fundamental to AI visibility because large language models default to what they know. Even when they query Google and elsewhere, LLMs often use their training data to guide the search terms.

It’s therefore essential to track what LLMs retain about your brand and competitors and, importantly, what is incorrect or outdated. Then focus on providing missing or corrected data on your site and across all owned channels.

Manual prompting in ChatGPT, Claude, and Gemini (at least) will help identify the gaps. The prompts can be:

  • “What do you know about [MY PRODUCT]?”
  • “Compare [MY PRODUCT] vs [MY COMPETITOR’S PRODUCT].”

Profound, Peec AI, and other AI visibility trackers can set up these prompts to monitor product positioning over time.

When using such visibility tools, keep in mind:

  • AI tracking tools enter prompts via LLMs’ APIs. Humans often see different results due to personalization and differences among AI models. API results are better for checking training data because LLMs likely return results from that data (versus live searches) to save resources.
  • The tools’ visibility scores depend entirely on the prompt. In the tools, separate branded prompts in a folder, as they will likely score 100%. Also, focus on non-branded prompts that reflect a product’s value proposition. Prompts irrelevant to an item’s key features will likely score 0%.

Most cited sources

LLM platforms increasingly conduct live searches when responding to prompts. They may query Google or Bing — yes, organic search drives AI visibility — or crawl other sources such as Reddit.

Citations, such as articles or videos, from those live searches influence the AI responses. But the citations vary widely because LLMs fan out across different (often unrelated) queries. So, trying to get included in every cited source is not realistic.

However, prompts often produce the same, influential sources repeatedly. These are worth exploring to include your brand or product. AI visibility trackers can collect the most cited URLs for your brand, product, or industry.

Brand mentions and branded search volume

Use Search Console or other traditional analytics tools to track:

  • Queries that contain your brand name or a version of it.
  • Number of clicks from those queries.
  • Impressions from those queries. The more AI answers include a brand name, the more humans will search for it.

In Search Console, create a filter in the “Performance” section to view data for branded queries.

Screenshot of the Search Console Performance section.

Create a filter in Search Console’s “Performance” section to view data for branded queries.