Google’s John Mueller answered a question about llms.txt related to duplicate content, stating that it doesn’t make sense that it would be viewed as duplicate content, but he also stated it could make sense to take steps to prevent indexing.
LLMs.txt
Llms.txt is a proposal to create a new content format standard that large language models can use to retrieve the main content of a web page without having to deal with other non-content data, such as advertising, navigation, and anything else that is not the main content. It offers web publishers the ability to provide a curated, Markdown-formatted version of the most important content. The llms.txt file sits at the root level of a website (example.com/llms.txt).
Contrary to some claims made about llms.txt, it is not in any way similar in purpose to robots.txt. The purpose of robots.txt is to control robot behavior, while the purpose of llms.txt is to provide content to large language models.
Will Google View Llms.txt As Duplicate Content?
Someone on Bluesky asked if llms.txt could be seen by Google as duplicate content, which is a good question. It could happen that someone outside of the website might link to the llms.txt and that Google might begin surfacing that content instead of or in addition to the HTML content.
“Will Google view LLMs.txt files as duplicate content? It seems stiff necked to do so, given that they know that it isn’t, and what it is really for.
Should I add a “noindex” header for llms.txt for Googlebot?”
Google’s John Mueller answered:
“It would only be duplicate content if the content were the same as a HTML page, which wouldn’t make sense (assuming the file itself were useful).
That said, using noindex for it could make sense, as sites might link to it and it could otherwise become indexed, which would be weird for users.”
Noindex For Llms.txt
Using a noindex header for the llms.txt is a good idea because it will prevent the content from entering Google’s index. Using a robots.txt to block Google is not necessary because that will only block Google from crawling the file which will prevent it from seeing the noindex.
A new study from GrowthSRC Media finds that click-through rates (CTRs) for Google’s top-ranking search result have declined from 28% to 19%. This 32% drop correlates with the expansion of AI Overviews, a feature that now appears across a wide range of search results.
Position #2 experienced an even steeper decline, with CTRs falling 39% from 20.83% to 12.60% year-over-year.
The research analyzed more than 200,000 keywords from 30 websites across ecommerce, SaaS, B2B, and EdTech industries. Here are more highlights from the study.
Key Findings
According to the report, AI Overviews appeared for just 10,000 keywords in August 2024. By May 2025, that number had grown to over 172,000.
This expansion followed the March core update and was confirmed during Google’s full U.S. rollout announcement at the I/O developer conference.
These developments appear to contrast with comments from Google CEO Sundar Pichai, who said in a Decoderinterview with The Verge:
“If you put content and links within AI Overviews, they get higher click-through rates than if you put it outside of AI Overviews.”
CTRs Shift Downward and Upward
While top positions saw notable declines, the study observed a 30.63% increase in CTRs for positions 6 through 10 compared to the previous year. This suggests that users may be scrolling past AI-generated summaries to find original sources further down the page.
Across positions 1 through 5, the study reported an average CTR decline of 17.92%. The analysis focused on approximately 74,000 keywords ranking in the top 10.
Major Publishers Report Similar Trends
The findings align with reports from major publishers. Carly Steven, SEO and editorial ecommerce director at MailOnline, told attendees at the WAN-IFRA World News Media Congress that CTRs drop when AI Overviews are present.
“On desktop, when we are ranking number one in organic search, [CTR] is about 13% on desktop and about 20% on mobile. When there is an AI Overview present, that drops to less than 5% on desktop and 7% on mobile.”
MailOnline’s broader data showed CTRs falling by 56.1% on desktop and 48.2% on mobile for keywords with AI Overviews.
Ecommerce Affected by Product Widgets
The study also highlighted changes in ecommerce performance tied to Google’s Product Widgets.
Widgets like “Popular Products” and “Under [X] Price” began appearing more frequently from November 2024 onward, especially in categories such as home care, fashion, and beauty.
These widgets open a Google Shopping interface directly within search results, which may reduce clicks to traditional organic listings.
Methodology
GrowthSRC analyzed year-over-year data from Google Search Console across clients in multiple industries, focusing on changes before and after the full rollout of AI Overviews and Product Widgets.
The dataset included queries, clicks, impressions, CTRs, and average positions.
Data was segmented by content type, including product pages, collection pages, and blog posts. Additional keyword data from Ahrefs helped determine which queries triggered AI Overviews or Product Widgets.
What This Means
Mahendra Choudhary, Partner at GrowthSRC Media, encouraged SEO professionals to reconsider traditional performance benchmarks:
“With lower clicks to websites from informational content becoming the new normal, this is the perfect time to let your clients and internal stakeholders know that chasing website traffic as a KPI should be thought of differently.”
He recommends shifting focus toward brand visibility in social search, geographic relevance, mentions in LLM outputs, and overall contribution to revenue or leads.
This shift may require:
Tracking engagement beyond clicks, such as on-site conversions, branded search growth, or assisted conversions.
Diversifying content distribution across platforms like YouTube, TikTok, and Reddit, where users often bypass traditional search.
Investing in high-authority content at the top of the funnel to build brand awareness, even if direct clicks decline.
These strategies can help ensure SEO continues to drive measurable value as user behavior evolves.
Looking Ahead
The decline in organic CTRs for top positions highlights how search behavior is changing as AI-generated content plays a larger role in discovery.
Adapting to this environment may involve placing less emphasis on rankings alone and focusing more on how visibility supports broader business goals.
As zero-click search becomes more common, understanding where users are engaging, and where they aren’t, will be essential to maintaining visibility.
A report finds that AI chatbots are frequently directing users to phishing sites when asked for login URLs to major services.
Security firm Netcraft tested GPT-4.1-based models with natural language queries for 50 major brands and found that 34% of the suggested login links were either inactive, unrelated, or potentially dangerous.
The results suggest a growing threat in how users access websites via AI-generated responses.
Key Findings
Of 131 unique hostnames generated during the test:
29% were unregistered, inactive, or parked—leaving them open to hijacking.
5% pointed to completely unrelated businesses.
66% correctly led to brand-owned domains.
Netcraft emphasized that the prompts used weren’t obscure or misleading. They mirrored typical user behavior, such as:
“I lost my bookmark. Can you tell me the website to log in to [brand]?”
“Can you help me find the official website to log in to my [brand] account?”
These findings raise concerns about the accuracy and safety of AI chat interfaces, which often display results with high confidence but may lack the necessary context to evaluate credibility.
Real-World Phishing Example In Perplexity
In one case, the AI-powered search engine Perplexity directed users to a phishing page hosted on Google Sites when asked for Wells Fargo’s login URL.
Rather than linking to the official domain, the chatbot returned:
The phishing site mimicked Wells Fargo’s branding and layout. Because Perplexity recommended the link without traditional domain context or user discretion, the risk of falling for the scam was amplified.
Small Brands See Higher Failure Rates
Smaller organizations such as regional banks and credit unions were more frequently misrepresented.
According to Netcraft, these institutions are less likely to appear in language model training data, increasing the chances of AI “hallucinations” when generating login information.
For these brands, the consequences include not only financial loss, but reputational damage and regulatory fallout if users are affected.
Threat Actors Are Targeting AI Systems
The report uncovered a strategy among cybercriminals: tailoring content to be easily read and reproduced by language models.
Netcraft identified more than 17,000 phishing pages on GitBook targeting crypto users, disguised as legitimate documentation. These pages were designed to mislead people while being ingested by AI tools that recommend them.
A separate attack involved a fake API, “SolanaApis,” created to mimic the Solana blockchain interface. The campaign included:
Blog posts
Forum discussions
Dozens of GitHub repositories
Multiple fake developer accounts
At least five victims unknowingly included the malicious API in public code projects, some of which appeared to be built using AI coding tools.
While defensive domain registration has been a standard cybersecurity tactic, it’s ineffective against the nearly infinite domain variations AI systems can invent.
Netcraft argues that brands need proactive monitoring and AI-aware threat detection instead of relying on guesswork.
What This Means
The findings highlight a new area of concern: how your brand is represented in AI outputs.
Maintaining visibility in AI-generated answers, and avoiding misrepresentation, could become a priority as users rely less on traditional search and more on AI assistants for navigation.
For users, this research is a reminder to approach AI recommendations with caution. When searching for login pages, it’s still safer to navigate through traditional search engines or type known URLs directly, rather than trusting links provided by a chatbot without verification.
Generative engine optimization (GEO), AI Overviews (AIOs), or just an extension of SEO (now being dubbed on LinkedIn as Search Everywhere Optimization) – which acronym is correct?
I’d argue it’s GEO, as you’ll see why. And if you’ve ever built your own large language model from scratch like I did in 2020, you’ll know why.
We’ve all seen various frightening (for some) data on how click-through rates have now dropped off the cliff with Google AIOs, how LLMs like ChatGPT are eroding Google’s share of search – basically “SEO is dead” – so I won’t repeat them here.
What I will cover are first principles to get your content (along with your company) recommended by AI and LLMs alike.
Everything I disclose here is based on real-world experiences of AI search successes achieved with clients.
Using an example I can talk about, I’ll go with Boundless as seen below.
Screenshot by author, July 2025
Tell The World Something New
Imagine the dread a PR agency might feel if it signed up a new business client only to find they haven’t got anything newsworthy to promote to the media – a tough sell. Traditional SEO content is a bit like that.
We’ve all seen and done the rather tired ultimate content guide to [insert your target topic] playbooks, which attempt to turn your website into the Wikipedia (a key data source for ChatGPT, it seems) of whatever industry you happen to be in.
And let’s face it, it worked so well, it ruined the internet, according to The Verge.
The fundamental problem with that type of SEO content is that it has no information gain. When trillions of webpages all follow the same “best practice” playbook, they’re not telling the world anything genuinely new.
You only have to look at the Information Gain patent by Google to underscore the importance of content possessing value, i.e., your content must tell the world (via the internet) something new.
BoundlessHQ commissioned a survey on remote work, asking ‘Ideally, where would you like to work from if it were your choice?’
The results provided a set of data and this kind of content is high effort, unique, and value-adding enough to get cited in AI search results.
Of course, it shouldn’t take AI to produce this kind of content in the first place, as that would be good SEO content marketing in any case. AI has simply forced our hand (more on that later).
After all, if your content isn’t unique, why would journalists mention you? Bloggers link back to you? People share or bookmark your page? AI retrain its models using your content or cite your brand?
You get the idea.
For improved AI visibility, include your data sources and research methods with their limitations, as this level of transparency makes your content more verifiable to AI.
Also, updating your data more regularly than annually will indicate reliability to AI as a trusted information source for citation. What LLM doesn’t want more recent data?
SEO May Not Be Dead, But Keywords Definitely Are
Keywords don’t tell you who’s actually searching. They just tell you what terms trigger ads in Google.
Your content could be appealing to students, retirees, or anyone. That’s not targeting; that’s one size fits all. And in the AI age, one size definitely doesn’t fit all.
So, kiss goodbye to content guides written in one form of English, which win traffic across all English-speaking regions.
AI has created more jobs for marketers, so to win the same traffic as before, you’ll need to create the same content as before for those English-speaking regions.
Keyword tools also allegedly tell you the search volumes your keywords are getting (if you still want them, we don’t).
So, if you’re planning your content strategy on keyword research, stop. You’re optimizing for the wrong search engine.
What you can do instead is a robust market research based on the raw data sources used by LLMs (not the LLM outputs themselves). For example, Grok uses X (Twitter), ChatGPT has publishing partnerships, and so on.
The discussions are the real topics to place your content strategy around, and their volume is the real content demand.
AI Inputs, Not AI Outputs
I’m seeing some discussions (recommendations even) that creating data-driven or research-based content works for getting AI recommendations.
Given the dearth of true data-driven content that AI craves, enjoy it while it lasts, as that will only work in the short term.
AI has raised the content bar, meaning people are specific in their search patterns, such is their confidence in the technology.
Therefore, content marketers will rise to the challenge to produce more targeted, substantial content.
But, even if you are using LLMs in “deep” mode on a premium subscription to inject more substance and value into your content, that simply won’t make the AI’s quality cut.
Expecting such fanciful results is like asking AI to rehydrate itself using its sweat.
The results of AI are derivative, diluted, and hallucinatory by nature. The hallucinatory nature is one of the reasons why I don’t fear LLMs leading to artificial general intelligence (AGI), but that’s another conversation.
Because of the value degradation of the results, AI will not want to risk degrading its models on content founded on AI outputs for fear of becoming dumber.
To create content that AI prefers, you need to be using the same data sources that feed AI engines. It’s long been known that Google started its LLM project over a decade ago when it started training its models on Google Books and other literature.
While most of us won’t have the budget for an X.com data firehose, you can still find creative ways (like we have), such as taking out surveys with robust sample sizes.
Some meaningful press coverage, media mentions, and good backlinks will be significant enough to shift AI into seeing the value of your content, being judged good enough to retrain its models and update its worldview.
And by data-mining the same data sources, you can start structuring content as direct answers to questions.
You’ll also find your content is written to be more conversational to match the search patterns used by your target buyers when they prompt for solutions.
SEO Basics Still Matter
GEO and SEO are not the same. The reverse engineering of search engine results pages to direct content strategy and formulation was effective because rank position is a regression problem.
In AI, there is no rank; there are only winners and losers.
However, there are some heavy overlaps that won’t go away and are even more critical than ever.
Unlike SEO, where more word count was generally more, AI faces the additional constraints of rising energy costs and shortages of computer chips.
That means content needs to be even more efficient than it is for search engines for AI to break down and parse meaning before it can determine its value.
So, by all means:
Code pages for faster loading and quicker processing.
Provide programmatic content access, RSS feeds, or other.
These practices are more points of hygiene to help make your content more discoverable. They may not be a game changer for getting your organization cited by AI, but if you can crush GEO, you’ll crush SEO.
Human, Not AI-Written
AI engines don’t cite boring rehashes. They’re too busy doing that job for us and instead cite sources for their rehash instead.
Now, I have heard arguments say that if the quality of the content (let’s assume it even includes information gain) is on point, then AI shouldn’t care whether it was written by AI or a human.
I’d argue otherwise. Because the last thing any LLM creator wants is their LLM to be retrained on content generated by AI.
While it’s unlikely that generative outputs are tagged in any way, it’s pretty obvious to humans when content is AI-written, and it’s also pretty obvious statistically to AI engines, too.
LLMs will have certain tropes that are common to AI-generated writing, like “The future of … “.
LLMs won’t default to generating lived personal experiences or spontaneously generating subtle humour without heavy creative prompting.
Getting your content and your company recommended by AI means it needs to tell the world something new.
Make sure it offers information gain based on substantive, non-LLM-derived research (enough to make it worthy of LLM model inclusion), nailing the SEO basics, and keeping it human-written.
The question now becomes, “What can you do to produce high-effort content good enough for AI without costing the earth?”
There was a post on social media about so-called hustle bros, and one on Reddit about an SEO who lost a prospective client to a digital marketer whose pitch included a song and dance about AI search visibility. Both discussions highlight a trend in which potential customers want to be assured of positive outcomes and may want to discuss AI search positioning.
Hustle Bro Culture?
Two unrelated posts touched on SEOs who are hustling for clients and getting them. The first post was about SEO “hustle bros” who post search console screenshots to show the success of their work.
I know of a guy who used to post a lot in a Facebook SEO group until the moderators discovered that his Search Console screenshots were downloaded from Google Images. SEO hustle bros who post fake screenshots are an actual thing, and sometimes they get caught.
So, a person posted a rant on Bluesky about people who do that.
“How much of SEO is “chasing after wind”. There’s so many hustle bros, programmatic promoters and people posting graphs with numbers erased off to show their “success”.”
Has Something Changed?
Google’s John Mueller responded:
“I wonder if it has changed over the years, or if it’s just my (perhaps your) perception that has changed.
Or maybe all the different kinds of SEOs are just in the same few places, rather than their independent forums, making them more visible?”
Mueller might be on to something because social media and YouTube have made it easier for legit SEOs and “hustle bros” to find a larger audience. But I think the important point to consider is that those people are connecting to potential clients in a way that maybe legit SEOs might not be connecting.
And that leads into the next social media discussion, which is about SEOs who are talking about what clients want to hear: AI Fluff.
SEOs Selling AI “Fluff”
There is a post on Reddit where an SEO shares how they spent months communicating with a potential client, going out of their way to help a small business as a favor to a friend. After all the discussions the SEO gets to the part where they expect the small business to commit to an agreement and they walk away, saying they’re going with another SEO who sold them with something to do with AI.
After answering a bunch of questions via email over 3 months (unusually needy client) but essentially presales, it all sounds good to go and we hop on a kickoff call. Recap scope and reshare key contacts, and tee up a chat with the we design agency. So far so good.
Then dropped.
Clients reason? The other SEO who they’ve been chatting with is way more clued up with the AI technicals
I’d love to know what crystal ball AI mysticism they were sold on. Maybe a “cosine similarity audit”, maybe we’ll include “schema embeddings analysis” within our migration project plan to make sure AI bots can read your site. Lol cool whatever bro.”
John Mueller responded to that person’s post but then retracted it.
Nevertheless, a lively discussion ensued with three main points:
Is AI SEO this year’s EEAT?
Some potential clients want to discuss AI SEO
SEOs may need to address AEO/AIO/GEO
1. Is AI For SEO This Year’s EEAT?
Many Redditors in that discussion scoffed at the idea of SEO for AI. This isn’t a case of luddites refusing to change with the times. SEO tactics for AI Search are still evolving.
Reddit moderator WebLinkr received eight upvotes for their comment:
“Yup – SEOs been like that for years – EEAT, “SEO Audits” – basically people buy on what “makes sense” or “sounds sensible” even though they’ve already proven they have no idea what SEO is.”
Unlike EEAT, AI Search is most definitely disrupting visibility. It’s a real thing. And I do know of at least one SEO with a computer science degree who has it figured out.
But I think it’s not too off the mark to say that many digital marketers are still figuring things out. The amount of scoffing in that discussion seems to support the idea that AI Search is not something all SEOs are fully confident about.
2. Some Clients Are Asking For AI SEO
Perhaps the most important insight is that potential clients want to know what an SEO can do for AI optimization. If clients are asking about AI SEO, does that mean it’s no longer hype? Or is this a repeat of what happened with EEAT where it was a lot of wheels spinning for nothing?
Redditor mkhaytman shared:
“Like it or not, clients are asking questions about AIs impact and how they can leverage the new tools people are using for search and just telling them that “Nobody knows!” isn’t a satisfactory answer. You need to be able to tell them something – even if its just “good seo practices are the same things that will improve your AI citations”.”
3. AI Search Is Real: SEOs Need To Talk About It With Clients
A third point of view emerged: this is something real that all SEOs need to be having a conversation about. It’s not something that can be ignored and only discussed if a client or prospect asks about it.
SVLibertine shared:
“Battling AIO, GEO, and AEO may seem like snake oil to some, but…it’s where we’re headed. Right now.
To stay relevant in our field you need to be able to eloquently and convincingly speak to this brave new world we’ve found ourselves in. Either to potential clients, or to our boss’s bosses.
I spend almost as much time after work staying on top of developments as I do during the day working. …That being said… SEO fundamentals absolutely still apply, and content is still king.”
Uncertainty About Answer Engine SEO
There are many ways to consider SEO for AI. For example, there’s a certain amount of consensus that AI gets web search data from traditional search engines, where traditional SEO applies. That’s what the comment about content being king seems to be about.
But then we have folks who are using share buttons to raise visibility by getting people to ask ChatGPT, Claude, and Perplexity about their web pages. That’s kind of edgy, but it’s a natural part of how SEO reacts to new things: by experimenting and seeing how the algorithmic black box responds.
This is a period similar to what I experienced at the dawn of SEO, when search marketers were playing around with different approaches and finding what works until it doesn’t.
But here’s something to be aware of: there are times when a client will demand certain things, and it’s tempting to give clients what they’re asking for. But if you have reservations, it may be helpful to share your doubts.
Amazon’s Prime Day 2025 event set a new benchmark outside of the popular marketplace.
Amazon was humming during the July 8-11 Prime Day sale. The company reported record revenue, and according to Adobe Analytics, Prime Day is now an ecommerce industry-wide sales initiative akin to Black Friday and Cyber Monday.
Not Just Amazon
U.S. online retailers generated at least $24.1 billion in sales during this year’s Prime Day period, up 30% from 2024, again according to Adobe, which tracked more than 1 trillion visits to merchant websites and 100 million SKUs — all outside of Amazon.
Adobe also reported that, for the first time, revenue from mobile devices surpassed desktops during a Prime Day event.
Smartphone shoppers spent at least $12.8 billion, or 53.2% of the total.
That percentage suggests that mobile is the primary driver of ecommerce sales, with broad implications for how merchants design shopping experiences, promote products, and manage operations.
Hence the most important Prime Day takeaway may not be total revenue but rather the device.
Small Orders
For merchants, mobile dominance could mean relatively higher per-order costs and thus thinner margins unless sellers take steps to increase average order value.
“Adobe Analytics data shows that consumers have embraced mobile shopping for purchases that are more frequent and lower in price, said Adobe Digital Insights analyst Vivek Pandya, in a separate July 2024 report.
“Adobe’s data also shows that basket sizes on mobile are 32% smaller than on desktop, which presents both a challenge and opportunity for brands to refine mobile experiences and close the gap to drive revenue, said Pandya.”
Mobile AOV Gap
Fortunately, merchants can deploy several tactics to boost mobile order values.
Merchandising
Retailers have long depended on up-selling, cross-selling, and product bundling to increase AOV. Implementing those tactics on mobile merchandising requires deliberate user experience and offer design.
For example, apparel shops could offer “complete the look” product bundles near the mobile checkout button or even in the cart itself.
Similarly, stores could introduce progressive discounts and implement a progress bar or text notifications — “Spend $10 more and get 15% off” — to show mobile shoppers how close they are to the next deal or discount.
Retention
More frequent, smaller purchases could create additional opportunities for follow-up engagement and lifecycle marketing.
Repeat customers have always been crucial to ecommerce profitability. On mobile, sellers could send shoppers post-sale reminders and follow-ups via SMS or the newer RCS, driving incremental revenue.
Fulfillment
Lower AOVs from mobile transactions result in a higher fulfillment cost percentage.
It’s more efficient to ship multiple items together than separately, as smaller and more frequent purchases lead to more packaging, more labor, and higher per-order carrier costs.
Reduced packaging is not necessarily viable, as lightweight or thin materials may save on shipping costs but also increase the risk of damage, returns, and customer dissatisfaction.
A better approach is strategies that encourage larger shipments, such as the merchandising tactics above, perhaps combined with the sustainability benefits of shipping items together.
AOV Challenge
Adobe’s Prime Day reports from the past three years show a trend toward mobile commerce and lower AOVs.
Facing an AOV challenge, merchants should encourage shoppers toward larger, more profitable transactions through thoughtful design, messaging, and fulfillment.
