LLMs.txt For AI SEO: Is It A Boost Or A Waste Of Time? via @sejournal, @martinibuster

Many popular WordPress SEO plugins and content management platforms offer the ability to generate LLMs.txt for the purpose of improving visibility in AI search platforms. With so many popular SEO plugins and CMS platforms offering LLMs.txt functionality, one might come away with the impression that it is the new frontier of SEO. The fact, however, is that LLMs.txt is just a proposal, and no AI platform has signed on to use it.

So why are so many companies rushing to support a standard that no one actually uses? Some SEO tools offer it because their users are asking for it, while many users feel they need to adopt LLMs.txt simply because their favorite tools provide it. A recent Reddit discussion on this very topic is a good place to look for answers.

Third Party SEO Tool And LLMs.txt

Google’s John Mueller addressed the LLMs.txt confusion in a recent Reddit discussion.  The person asking the question was concerned because an SEO tool flagged it as 404, missing. The user had the impression that the tool implied it was needed.

Their question was:

“Why is SEMRush showing that the /llm.txt is a 404? Yes, I. know I don’t have one for the website, but, I’ve heard it’s useless and not needed. Is that true?

If i need it, how do i build it?

Thanks”

The Redditor seems to be confused by the Semrush audit that appears to imply that they need an LLMs.txt. I don’t know what they saw in the audit but this is what the official Semrush audit documentation shares about the usefulness of LLMs.txt:

“If your site lacks a clear llms.txt file it risks being misrepresented by AI systems.

…This new check makes it easy to quickly identify any issues that may limit your exposure in AI search results.”

Their documentation says that it’s a “risk” to not have an LLMs.txt but the fact is that there is absolutely no risk because no AI platform uses it. And that may be why the Redditor was asking the question, “If i need it, how do I build it?”

LLMs.txt Is Unnecessary

Google’s John Mueller confirmed that LLMs.txt is unnecessary.

He explained:

“Good catch! Especially in SEO, it’s important to catch misleading & bad information early, before you invest time into doing something unnecessary. Question everything.”

Why AI Platforms May Choose To Not Use LLMs.txt

Aside from John Mueller’s many informal statements about the uselessness of LLMs.txt, I don’t think there are any formal statements from AI platforms as to why they don’t use LLMs.txt and their associated .md markdown texts. There are, however, many good reasons why an AI platform would choose not to use it.

The biggest reason not to use LLMs.txt is that it is inherently untrustworthy. On-page content is relatively trustworthy because it is the same for users as it is for an AI bot.

A sneaky SEO could add things to structured data and markdown texts that don’t exist in the regular HTML content in order to get their content to rank better. It is naive to think that an SEO or publisher would not use .md files to trick AI platforms.

For example, unscrupulous SEOs add hidden text and AI prompts within HTML content. A research paper from 2024 (Adversarial Search Engine Optimization for Large Language Models) showed that manipulation of LLMs was possible using a technique they called Preference Manipulation Attacks.

Here’s a quote from that research paper (PDF):

“…an attacker can trick an LLM into promoting their content over competitors. Preference Manipulation Attacks are a new threat that combines elements from prompt injection attacks… Search Engine Optimization (SEO)… and LLM ‘persuasion.’

We demonstrate the effectiveness of Preference Manipulation Attacks on production LLM search engines (Bing and Perplexity) and plugin APIs (for GPT-4 and Claude). Our attacks are black-box, stealthy, and reliably manipulate the LLM to promote the attacker’s content. For example, when asking Bing to search for a camera to recommend, a Preference Manipulation Attack makes the targeted camera 2.5× more likely to be recommended by the LLM.”

The point is that if there’s a loophole to be exploited, someone will think it’s a good idea to take advantage of it, and that’s the problem with creating a separate file for AI chatbots: people will see it as the ideal place to spam LLMs.

It’s safer to rely on on-page content than on a markdown file that can be altered exclusively for AI. This is why I say that LLMs.txt is inherently untrustworthy.

What SEO Plugins Say About LLMs.txt

The makers of Squirrly WordPress SEO plugin acknowledge that they provided the feature only because their users asked for it, and they assert that it has no influence on AI search visibility.

They write:

“I know that many of you love using Squirrly SEO and want to keep using it. Which is why you’ve asked us to bring this feature.

So we brought it.

But, because I care about you:

– know that LLMs txt will not help you magically appear in AI search. There is currently zero proof that it helps with being promoted by AI search engines.”

They strike a good balance between giving users what they want while also letting them know it’s not actually needed.

While Squirrly is at one end saying (correctly) that LLMs.txt doesn’t boost AI search visibility, Rank Math is on the opposite end saying that AI chatbots actually use the curated version of the content presented in the markdown files.

Rank Math is generally correct in its description of what an LLMs.txt is and how it works, but it overstates the usefulness by suggesting that AI chatbots use the curated LLMs.txt and the associated markdown files.

They write:

“So when an AI chatbot tries to summarize or answer questions based on your site, it doesn’t guess—it refers to the curated version you’ve given it. This increases your chances of being cited properly, represented accurately, and discovered by users in AI-powered results.”

We know for a fact that AI chatbots do not use a curated version of the content. They don’t even use structured data; they just use the regular HTML content.

Yoast SEO is a little more conservative, occupying a position in the center between Squirrly and Rank Math, explaining the purpose of LLMs.txt but not overstating the benefits by hedging with words like “can” and “could.” That is a fair way to describe LLMs.txt, although I like Squirrly’s approach that says, you asked for it, here it is, but don’t expect a boost in search performance.

The LLMs.txt Misinformation Loop

The conversation around LLMs.txt has become a self-reinforcing loop: business owners and SEOs feel anxiety over AI visibility and feel they must do something, viewing LLMs.txt as the something they can do.

SEO tool providers are compelled to provide the LLMs.txt option, reinforcing the belief that it’s a necessity, unintentionally perpetuating the cycle of misunderstanding.

Concern over AI visibility has led to the adoption of LLMs.txt which at this stage is only a proposal for a standard that no AI platform currently uses.

Featured Image by Shutterstock/James Delia

Google Answers SEO Question About Keyword Cannibalization via @sejournal, @martinibuster

Google’s John Mueller answered a question about a situation where multiple pages were ranking for the same search queries. Mueller affirmed the importance of reducing unnecessary duplication but also downplayed keyword cannibalization.

What Is Keyword/Content Cannibalization?

There is an idea that web pages will have trouble ranking if multiple pages are competing for the same keyword phrases. This is related to the SEO fear of duplicate content. Keyword cannibalization is just a catchall phrase that is applied to low-ranking pages that are on similar topics.

The problem with saying that something is keyword cannibalization is that it does not identify something specific about the content that is wrong. That is why there are people asking John Mueller about it, simply because it is an ill-defined and unhelpful SEO concept.

SEO Confusion

The SEO was confused about the recent &num=100 change, where Google is blocking rank trackers from scraping the search results (SERPs) at the rate of 100 results at a time. Some rank trackers are floating the idea of only showing ranking data for the top 20 search results. This affects rank trackers’ ability to scrape the SERPs and has no effect on Google Search Console other than to show more accurate results.

The SEO was under the wrong impression that Search Console was no longer showing impressions from results beyond the top twenty. This is false.

Mueller didn’t address that question; it is just a misunderstanding on the part of the SEO.

Here is the question that was asked:

“If now we are not seeing data from GSC from positions 20 and over, does that mean in fact there are no pages ranking above those places?

If I want to avoid cannibalization, how would I know which pages are being considered for a query, if I can only see URLs in the top 20 or so positions?”

Different Pages Ranking For Same Query

Mueller said that different pages ranking for the same search query is not a problem. I agree: multiple web pages ranking for the same keyword phrases is not a problem; it’s a good thing.

Mueller explained:

“Search Console shows data for when pages were actually shown, it’s not a theoretical measurement. Assuming you’re looking for pages ranking for the same query, you’d see that only if they were actually shown. (IMO it’s not really “cannibalization” if it’s theoretical.)

All that said, I don’t know if this is actually a good use of time. If you have 3 different pages appearing in the same search result, that doesn’t seem problematic to me just because it’s “more than 1″. You need to look at the details, you need to know your site, and your potential users.

Reduce unnecessary duplication and spend your energy on a fantastic page, sure. But pages aren’t duplicates just because they happen to appear in the same search results page. I like cheese, and many pages could appear without being duplicates: shops, recipes, suggestions, knives, pineapple, etc.”

Actual SEO Problems

Multiple pages ranking for the same keyword phrases is not a problem; it’s a good thing and not a reason for concern. Multiple pages not ranking for keywords is a problem.

Here are some real reasons why pages on the same topic may fail to rank:

• The pages are too long and consequently are unfocused.
• The pages contain off-topic passages.
• The pages are insufficiently linked internally.
• The pages are thin.
• The pages are virtually duplicates of the other pages in the group.

The above are just a few real reasons why multiple pages on the same topic may not be ranking. Pointing at the pages and declaring they are cannibalizing each other is not real. It’s not something to worry about because keyword cannibalization is just a catchall phrase that masks all the actual reasons I just listed.

Takeaway

The debate over keyword cannibalization says less about Google’s algorithm and more about how the SEO community is willing to accept ideas without really questioning whether the underlying basis makes sense. The question about keyword cannibalization is frequently discussed, and I think that’s because many SEOs have the intuition that it’s somehow not right.

Maybe the habit of diagnosing ranking issues with convenient labels mirrors the human tendency to prefer simple explanations over complex answers. But, as Mueller reminds us, the real story is not that two or three pages happen to surface for the same query. The real story is whether those pages are useful, well linked, and focused enough to meet a reader’s information needs.

What is diagnosed as “content cannibalization” is more likely something else. So, rather than chasing shadows, it may be better to look at the web pages with the eyes of a user and really dig into what’s wrong with the page or the interlinking patterns of the entire section that is proving problematic. Keyword cannibalization disappears the moment you look closer, and other real reasons become evident.

Featured Image by Shutterstock/Roman Samborskyi

Google Confirms August Spam Update Is Complete via @sejournal, @MattGSouthern

Google says the August 2025 spam update has finished rolling out worldwide.

• The rollout ran from August 26 to September 22, and applied globally in all languages.
• Early third-party data points to muted effects with limited broad re-ranking.
• If you saw declines, review Google’s spam policies.

SEO For Paws Live Stream Conference: Free Tickets Out Now via @sejournal, @theshelleywalsh

The next SEO For Paws will be held on Sept. 25, 2025.

The live stream features a stellar speaker list that includes some of the industry’s best SEO professionals and personalities, including Andrey Lipattsev, David Carrasco, Judith Lewis, and Jamie Indigo.

SEO for Paws, is a live-streamed fundraiser founded by Anton Shulke, an expert at organizing events, to help a charity close to his heart.

Anton has tirelessly continued his support for his favorite charity, which aids the many pets that were left behind in Kyiv after war broke out in Ukraine. The previous event in March managed to generate approx $7,000 for the worthy cause, with all funds going straight to the shelters where it’s needed.

Anton is well-known for his love of cats. Dynia, who traveled across Europe with Anton’s family after escaping Kyiv, is a regular feature on his social media channels.

Image from Anton Shulke, September 2025

One Cat Turned Into A Shelter Of 50

Among the many pet shelters that SEO For Paws has helped is an apartment run by Alya, who cares for up to 50 animals.

Alya has always cared for animals, and meeting an old, sick cat she called Fox was the start of becoming an organized shelter.

In 2016, she started with five cats living in her apartment, and today has 50 alongside 15 of her grandmother’s cats.

There’s a lot involved in care for this many animals, including the feeding, cleaning, washing litter boxes, replacing litter, and performing hygiene or medical procedures when needed.

Running a home-based shelter is not easy. Sometimes it’s sad, sometimes it’s exhausting. But Alya says that looking around at all the little whiskered faces, the furry bodies sprawled across the furniture, makes it worth it. Giving them a life of warmth, food, and love is worth every challenge.

To keep supporting individuals like Alya, we need your help. You can donate via Anton’s Buy Me a Coffee.

SEO For Paws – Cat Lovers, Dog Lovers, And SEO

The upcoming “SEO for Paws” livestream aims to continue fundraising efforts. The event, which runs from 12:00 p.m. to 4:30 p.m. ET, will offer actionable SEO and digital marketing advice from experts while raising money for the animal shelters.

Headline speakers who have donated their time to support his cause include Andrey Lipattsev, David Carrasco, Olga Zarr, Judith Lewis, James Wirth, Zach Chahalis, Jamie Indigo, and Lee Elliott.

Attendance is free, but participants are encouraged to donate to help the charity.

Event Highlights

• Date and Time: September 25, 2025, from 12:00 p.m. to 4:30 p.m. ET.
• Access: Free registration with the option to join live, participate in Q&A sessions, and a recording will be made available on YouTube.
• Speakers: The live stream will feature SEO and digital marketing experts, who will share actionable insights.

How To Make A Difference

The “SEO for Paws” live stream is an opportunity to make a meaningful difference while listening to excellent speakers.

All money raised is donated to help cats and dogs in Ukraine.

You can register for the event here.

And you can help support the charity by buying coffee.

Search Engine Journal is proud to be sponsoring the event.

More Resources:

Featured Image: Anton Shulke/SEO For Paws

The Future Of Rank Tracking Can Go Two Ways via @sejournal, @martinibuster

Digital marketers are providing more evidence that Google’s disabling of the num=100 search parameter correlates exactly with changes in Google Search Console impression rates. What looked like reliable data may, in fact, have been a distorted picture shaped by third-party SERP crawlers. It’s becoming clear that squeezing meaning from the top 100 search results is increasingly a thing of the past and that this development may be a good thing for SEO.

Num=100 Search Parameter

Google recently disabled the use of a search parameter that caused web searches to display 100 organic search results for a given query. Search results keyword trackers depended on this parameter for efficiently crawling Google’s search results. By eliminating the search parameter, Google is forcing data providers into an unsustainable position that requires them to scale their crawling by ten times in order to extract the top 100 search results.

Rank Tracking: Fighting To Keep It Alive

Mike Roberts, founder of SpyFu, wrote a defiant post saying that they will find a way to continue bringing top 100 data to users.

His post painted an image of an us versus them moment:

“We’re fighting to keep it alive. But this hits hard – delivering is very expensive.

We might even lose money trying to do this… but we’re going to try anyway.

If we do this alone, it’s not sustainable. We need your help.

This isn’t about SpyFu vs. them.

If we can do it – the way the ecosystem works – all your favorite tools will be able to do it. If nothing else, then by using our API (which has 100% of our keyword and ranking data).”

Rank Tracking: Where The Wind Is Blowing

Tim Soulo, CMO of Ahrefs, sounded more pragmatic about the situation, tweeting that the future of ranking data will inevitably be focused on the Top 20 search results.

Tim observed:

“Ramping up the data pulls by 10x is just not feasible, given the scale at which all SEO tools operate.

So the question is:

‘Do you need keyword data below Top 20?’

Because most likely it’s going to come at a pretty steep premium going forward.

Personally, I see it this way:

▪️ Top 10 – is where all the traffic is at. Definitely a must-have.

▪️ Top 20 – this is where “opportunity” is at, both for your and your competitors. Also must-have.

▪️ Top 21-100 – IMO this is merely an indication that a page is “indexed” by Google. I can’t recall any truly actionable use cases for this data.”

Many of the responses to his tweet were in agreement, as am I. Anything below the top 20, as Tim suggested, only tells you that a site is indexed. The big picture, in my opinion, is that it doesn’t matter whether a site is ranked in position 21 or 91; they’re pretty much equivalently suffering from serious quality or relevance issues that need to be worked out. Any competitors in that position shouldn’t be something to worry about because they are not up and coming; they’re just limping their way in the darkness of page three and beyond.

Page two positions, however, provide actionable and useful information because they show that a page is relevant for a given keyword term but that the sites ranked above it are better in terms of quality, user experience, and/or relevance. They could even be as good as what’s on page one but, in my experience, it’s less about links and more often it’s about user preference for the sites in the top ten.

Distorted Search Console Data

It’s becoming clear that search results scraping distorted Google’s Search Console data. Users are reporting that Search Console keyword impression data is significantly lower since Google blocked the Num=100 search parameter. Impressions are the times when Google shows a web page in the search results, meaning that the site is ranking for a given keyword phrase.

SEO and web developer Tyler Gargula (LinkedIn profile) posted the results of an analysis of over three hundred Search Console properties, showing that 87.7% of the sites experienced drops in impressions. 77.6% of the sites in the analysis experienced losses in query counts, losing visibility for unique keyword phrases.

Tyler shared:

“Keyword Length: Short-tail and mid-tail keywords experienced the largest drops in impressions, with single word keywords being much lower than I anticipated. This could be because short and mid-tail keywords are popular across the SEO industry and easier to track/manage within popular SEO tracking tools.

Keyword Ranking Positions: There has been reductions in keywords ranking on page 3+, and in turn an increase in keywords ranking in the top 3 and page 1. This suggests keywords are now more representative of their actual ranking position, versus receiving skewed positions from num=100.”

Google Is Proactively Fighting SERP Scraping

Disabling the num=100 search parameter is just the prelude to a bigger battle. Google is hiring an engineer to assist in statistical analysis of SERP patterns and to work together with other teams to develop models for combating scrapers. It’s obvious that this activity negatively affects Search Console data, which in turn makes it harder for SEOs to get an accurate reading on search performance.

What It Means For The Future

The num=100 parameter was turned off in a direct attack on the scraping that underpinned the rank-tracking industry. Its removal is forcing the search industry to reconsider the value of data beyond the top 20 results. This may be a turning point toward better attribution and and clearer measures of relevance.

Featured Image by Shutterstock/by-studio

Google Is Hiring An Anti-Scraping Engineering Analyst via @sejournal, @martinibuster

Google is hiring a new anti-scraping czar, whose job will be to analyze search traffic to identify the patterns of search scrapers, assess the impact, and work with engineering teams to develop new anti-scraping models for improving anti-scraping defenses.

Search Results Scraping

SEOs rely on SERP tracking companies to provide search results data for understanding search ranking trends, enabling competitive intelligence, and other keyword-related research and analysis.

Many of these companies conduct massive amounts of automated crawling of Google’s search results to take a snapshot of ranking positions and data related to search features triggered by keyword phrases. This scraping is suspected of causing significant changes to what’s reported in Google Search Console.

In the early days of SEO, there used to be a free keyword data source via Yahoo’s Overture, their PPC service. Many SEOs used to search on Yahoo so often that their searches would unintentionally inflate the keyword volume. Smart SEOs would know better to not optimize for those keyword phrases.

I have suspected that some SEOs may also have intentionally scraped Yahoo’s search results using fake keyword phrases in order to generate keyword volumes for those queries, in order to mislead competitors into optimizing for phantom search queries.

&num=100 Results Parameter

There is a growing suspicion backed by Google Search Console data that search result scraping may have inflated the official keyword impression data and that it may be the reason why Search Console Data appears to show that AI Search results aren’t sending traffic while Google’s internal data shows the opposite.

This suspicion is based on falling keyword impressions that correlate with Google’s recent action to block generating 100 search results with one search query, a technique used by various keyword tracking tools.

Google Anti-Scraping Engineering Analyst

Jamie Indigo posted that Google is looking to hire an Engineering Analyst focused on combatting search scraping.

The responsibilities for the job are:

• “Investigate and analyze patterns of abuse on Google Search, utilizing data-motivated insights to develop countermeasures and enhance platform security.
  Analyze datasets to identify trends, patterns, and anomalies that may indicate abuse within Google Search.
• Develop and track metrics to measure scraper impact and the effectiveness of anti-scraping defenses. Collaborate with engineering teams to design, test, and launch new anti-scraper rules, models, and system enhancements.
• Investigate proof-of-concept attacks and research reports that identify blind spots and guide the engineering team’s development priorities. Evaluate the effectiveness of existing and proposed detection mechanisms, understanding the impact on scrapers and real users.
• Contribute to the development of signals and features for machine learning models to detect abusive behavior. Develop and maintain threat intelligence on scraper actors, motivations, tactics and the scraper ecosystem.”

What Does It Mean?

There hasn’t been an official statement from Google but it’s fairly apparent that Google may be putting a stop to search results scrapers. This should result in more accurate Search Console data, so that’s a plus.

Featured Image by Shutterstock/DIMAS WINDU