Google Must Give Notice Before Significant Ranking Changes via @sejournal, @MattGSouthern

The UK’s Competition and Markets Authority has introduced two new conduct requirements for Google’s general search services, one covering how organic results are ranked and another covering search data portability.

Under the fair ranking requirement, Google must rank organic results using objective and non-discriminatory criteria, including in AI Overviews, but not sponsored results. It also has to be more transparent about how rankings work, give advance notice of significant changes, and set up a process for businesses to raise concerns.

Fair Ranking

Businesses told the CMA that Google’s ranking practices are neither fair nor transparent, that changes arrive without sufficient notice, and that they have no effective way to raise concerns when those changes hurt them.

Will Hayter, Executive Director for Digital Markets at the CMA, said in the announcement:

“These new measures will ensure search results are ranked fairly and objectively, with clearer information about changes and effective routes to raise concerns.”

Google pushed back on the premise behind the ranking requirement. A spokesperson told Press Gazette that the company’s “ranking systems are fair, transparent and show the most relevant, highest quality results.”

Data Portability

The second requirement turns Google’s voluntary UK Data Portability API into a legal obligation. The tool already lets people share their search data with third-party services.

Those services want to build products around the data but have lacked reliable access, for uses like personalized shopping deals or cashback rewards. The requirement brings UK users’ data rights in line with the EU’s under the Digital Markets Act.

Timeline & Oversight

Google has six months to implement the fair ranking requirement and three months for data portability. The CMA will monitor compliance through regular reporting and may add further measures.

How We Got Here

The requirements follow the CMA’s early-June action that gave websites more control over whether their content is used to power Google’s AI features. Both sit under the UK’s digital markets competition regime, created by the Digital Markets, Competition and Consumers Act.

Google was designated with strategic market status in general search and search advertising last year. That designation is not a finding that Google broke competition law.

Why This Matters

The fair ranking requirement targets a long-standing frustration in search. UK businesses rely on Google Search to reach customers, but say the way rankings work is too unpredictable to plan around. Objective criteria, advance notice of significant changes, and a complaints process would give those businesses a defined route to raise concerns.

The requirement covers organic results, including those in AI Overviews, but not sponsored results. That places AI Overview ranking under the same fairness and transparency obligations as standard organic ranking.

The requirement doesn’t make Google’s ranking systems public. It sets obligations around criteria, notice, and complaints, not disclosure of the ranking algorithm.

Looking Ahead

The CMA is acting in stages, with more activity on Google’s search business expected over the summer. Both requirements apply only in the UK.

The open question now is implementation. The requirement’s value depends on how Google puts it into practice, and whether that satisfies the CMA.

The UK action adds to broader regulatory scrutiny of Google Search in other markets, including the United States and the European Union.

Featured Image: Tupungato/Shutterstock

Google Explains Why URLs Blocked By Robots.txt Can Still Be Indexed via @sejournal, @martinibuster

Google’s John Mueller answered a question about the curious circumstance of Search Console reporting thousands of URLs as indexed despite being blocked by robots.txt. Mueller helped explain how this happens and what to do about it.

Content Indexed Despite Being Blocked By Robots.txt

A Redditor asked for advice because Google Search Console was reporting more than 51,000 pages under the status “Indexed, though blocked by robots.txt.” The affected URLs were primarily WooCommerce product URLs containing add-to-cart URL parameters like “?add-to-cart=”.

Because the issue appeared suddenly, the site owner questioned whether the robots.txt rules themselves were responsible for creating the problem. They also wanted to know whether removing the rules would help Google process the canonical signals and eliminate the reported URLs from Search Console.

The person asked:

“I have WooCommerce site and suddenly since past month we are facing this issue: “Indexed, though blocked by robots.txt”

there are total “Affected pages 51K pages”

in the end of url I see mostly ?page&post_type=product&product=slug&add-to-cart=98063,

After inspecting those urls I found they have index tag setup and robots.txt had

* Disallow: /*?add-to-cart=
* Disallow: /*?*add-to-cart=

I removed those two rules from robots.txt and hoping those pages fixed cause they have canonical set to correct product, will that fix issue?

or should I also setup noindex rules? will that cause us our crawl budget? it is pretty big woocommerce site, let me know guys your thoughts if someone has experience fixing such issue? and what will be the right method without preventing our SEO or functionality loss.”

Google Says Add-To-Cart URLs Don’t Need To Be Indexed

Mueller responded that the add-to-cart URLs do not need to be indexed and that blocking them through robots.txt is an acceptable approach.

He explained that even when Google reports those URLs as indexed, they are unlikely to appear in normal search results because they are blocked by robots.txt. According to Mueller, users generally do not search for those URLs directly, making them poor candidates for search visibility.

John Mueller responded:

“You don’t need the add-to-cart URLs indexed. Blocking them with robots.txt is fine. Even if they get “indexed” since they’re blocked by robots.txt, it’s unlikely that they’ll be shown in search (unless you do specific queries for those URLs, which users don’t do).”

I’m kind of on the fence about what Mueller said about “robots.txt” making it “unlikely” that the URLs will be shown in Search. The reason is because robots.txt does not prevent a web page from showing in Google Search. It just prevents Googlebot from crawling those pages. So technically, that’s not quite correct and I’m a little surprised Mueller would say that.

Noindex Is Probably Not A Solution

One of the Redditors who responded to that question suggested the solution of adding a noindex robots tag to the parameterized URLs. But that may not be a viable solution because the pages with and without the URL parameters are essentially the same thing. They’re rendered using the same template for a specific page. So unless WooCommerce treats them differently and can render the parameterized URLs with a noindex and the regular page without the noindex, that’s not a real solution.

Why Google Reports Indexed URLs That It Can’t Crawl

Another Redditor offered a possible explanation for why so many URLs appeared in Search Console. They suggested that Google likely discovered links containing the add-to-cart parameters somewhere on the site and added those URLs to its systems.

My suggestion for the person who originally asked that question is to crawl the website with Screaming Frog, review the internal linking to identify where those pages are being linked from, and then take some action, like removing those links or adding a rel=”nofollow” link attribute to them.

Likely, the best solution is to use the robots.txt block to prevent crawling, as long as it’s understood that this is all it does. If the person wants to be extra sure, they can also identify where those links exist and then add the nofollow link attribute as an extra layer, a hint to Google. Nofollow is not a directive, but it is a strong hint.

Search Console Warnings Don’t Always Indicate A Search Problem

One of the recurring challenges with Search Console reports is that they can expose technical conditions that look distressing but actually have little to zero effect on search performance. For example, the 404 error reports are useful for a variety of reasons, but many times a 404 server response is the right response, and it’s not really an “error” that needs fixing.

Takeaway

Mueller’s response reinforces the takeaway that not every Search Console warning requires taking action to fix something, although in this specific case there may be something to fix in the form of internal links to webpages that use the shopping cart URL parameters. If those links with the shopping cart URL parameters are absolutely necessary, then using a rel=”nofollow” link attribute will give Google a strong hint not to follow that link. The joy of technical SEO!

Featured Image by Shutterstock/Orange Line Media

Google’s Updated Guidance Now Says It’s “Fine” To Use LLMs.txt For AI SEO via @sejournal, @martinibuster

Google updated its guidance on Generative AI SEO to lighten up on its previous guidance that discouraged the use of LLMs.txt and other forms of markup and markdown. The new guidance strikes a more balanced tone that acknowledges there are AI search surfaces other than Google’s that users may want to optimize for.

LLMs.txt Guidance

Google’s guidance originally recommended that LLMs.txt and other kinds of special markup are not needed in order to rank in generative AI search, which was a broad statement that likely unintentionally encompassed all generative AI search surfaces, not just Google’s.

The new guidance tightens that passage to make it clear that the statement that special markup is not necessary for website optimization is limited to “Google Search (including its generative AI capabilities).”

Google Says Specials Markups Are Fine To Use

A similar update is a change in the guidance so that it no longer discourages SEOs and site owners from using LLMs.txt and other tactics like markdown for LLMs. Now it simply states that Google doesn’t use them but that if people want to use them, then go ahead.

The updated guidance now says:

“It’s completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so won’t harm (nor help) your visibility or rankings in Google Search, as Google Search ignores them.”

Guidance Is The Same But Improved

Google’s updated Search Central web page is explicitly about optimizing websites for “generative AI features on Google Search” so there was nothing technically wrong with the previous version. Yet this is an improvement because people tend to read and quote portions out of context which can give an unintended impression. This clear that up.

As Google’s changelog notes:

“Added a note to the AI optimization guide clarifying Google Search’s usage of llms.txt files.

Why: To address questions from the community and clarify that while these files aren’t needed for Google Search (and won’t negatively or positively impact your visibility or rankings), it’s fine if you want to maintain these files for other services or systems that use them.”

Featured Image by Shutterstock/Koshiro K

Google Says Markdown For AI SEO Strips Away The Parts That Matter via @sejournal, @martinibuster

On a recent Search Off the Record podcast, hosts John Mueller and Martin Splitt pushed back on the idea promoted by AI SEOs that stripped-down, content-only versions are a better way to optimize for AI Search. They made the case that all the things AI SEOs want to remove are actually useful for ranking.

Non-Content Parts Of Web Pages Matter

The TL;DR of this part is that HTML is for browsers to render into a visible page for humans, as well as for screen readers to read.

Martin Splitt begins the discussion by explaining why plain HTML appears not to be the ideal way to provide content to AI agents and LLMs. The idea is that, in addition to content, there’s a lot of other code in the HTML that is irrelevant for an LLM or AI agent that may be visiting a site for the content.

The appeal of markdown, then, is that it can provide the content in a manner that breaks free of all the HTML that’s meant to make a web page visible for humans or readable by a screen reader.

Splitt explains:

“And I think that’s also why people think it’s good for LLMs, because you have less stuff, less tokens. And if you look at an HTML file without a browser rendering it, if you just look at the plain HTML in a text editor, basically, then it’s hard to read the content, because there’s so much cruft, so much stuff in it. There’s all these HTML tags and all this maybe even inline styles and all that kind of stuff.”

He also praises markdown for the ability to still communicate the essence of the content:

“But if a Markdown render fails and you look at the Markdown file in a text editor, it still is structured and readable. Like a link is the word of the link text, like the anchor text, and then in square brackets and then in normal brackets. It’s probably what I would do if text was all I had available.

If I was writing an email without the possibility to actually link things, I would probably mark up some sort of link text and then put some sort of way to say, like, and this is where you need to go to actually see that.

And I think this minimalism is probably what makes people think, yeah, this is great for a machine that needs to understand this content, unlike HTML.”

Converting HTML To Text Is Trivial

Mueller and Splitt noted that despite how complex HTML looks, crawling and making sense of it is trivial and very easy to do. The selling point about using markdown for LLMs, that it simplifies crawling and indexing content, completely breaks down at this point.

John Mueller explains:

“I think the big thing is that the web with HTML and everything has been around for really long time, longer than Markdown. And all of the crawlers out there, have practiced with HTML. And converting HTML into text is trivial. There are lots of libraries out there that can do that for you. So if you think about what an average web crawler might look for or might need to find on a page to be able to understand it, then probably that’s just HTML.”

Markdown Fails For Content Discovery

Discovery is when any crawler visits a web page and discovers other web pages within a single website, and also from website to website.

Splitt said that markdown is focused on just one part of the content: the content itself. He explained that this makes it harder for search engines to see a web page in the context of how it connects to the rest of a website’s content through links, which aid discovery.

He explained:

“Yeah, and I mean, the other thing is, yes, it’s nice that Markdown is usually then focusing on a piece of content, but HTML with all the links and navigation and the headers and all that kind of stuff that kind of gets stripped out in the Markdown files that make the website are important to understand the structure and how this connects to the rest of the site.

So I guess that’s also a bad thing. If we were to lose this, that’s probably not so good for crawling in Discovery, huh? “

Takeaway

Reading patents and research papers, it becomes clear that search engines see a website as a collection of individual web pages, but also as groups of web pages that belong to sections and categories, and also as the entire website itself as a whole. Zoom out, and the website is but one point among thousands and thousands of other websites in a neighborhood of websites, self-organized by links into categories and quality levels.

For SEO, we have to understand a site from both the zoomed-out and zoomed-in view to conceptualize how all the pieces fit together. The reason is because that’s what search engines do.

AI-based SEO seems to be hung up on making it easy for LLMs and AI agents to crawl and index content. Crawling and indexing are valid concerns. But by insisting on markdown files, they are not considering the fundamentals of discovery and how trivial it is to extract content from an HTML web page, which makes markdown files redundant.

Aside from the above issues, there is also the one about trustworthiness. There used to be a thing called a keyword meta tag that some search engines used to get a hint about what a web page was about. Naturally, site owners and SEOs used it to dump all the keywords they wanted to rank for, regardless of the content.

I’m not saying that SEOs and website owners are untrustworthy, but search traffic is money, and people are going to do what they’re going to do. So the last consideration is that search engines will never trust markdown content and use it as the canonical when it’s a trivial thing to crawl and extract the original content from the HTML.

Circling back to what Mueller and Splitt discussed, Google insists that the AI SEO insistence on markdown strips away a significant amount of context that matters.

Watch Search Off The Record Episode 111 here: