Anthropic’s Claude Bots Make Robots.txt Decisions More Granular via @sejournal, @MattGSouthern

Anthropic updated its crawler documentation this week with a formal breakdown of its three web crawlers and their individual purposes.

The page now lists ClaudeBot (training data collection), Claude-User (fetching pages when Claude users ask questions), and Claude-SearchBot (indexing content for search results) as separate bots, each with its own robots.txt user-agent string.

Each bot gets a “What happens when you disable it” explanation. For Claude-SearchBot, Anthropic wrote that blocking it “prevents our system from indexing your content for search optimization, which may reduce your site’s visibility and accuracy in user search results.”

For Claude-User, the language is similar. Blocking it “prevents our system from retrieving your content in response to a user query, which may reduce your site’s visibility for user-directed web search.”

The update formalizes a pattern that’s becoming more common among AI search products. OpenAI runs the same three-tier structure with GPTBot, OAI-SearchBot, and ChatGPT-User. Perplexity operates a two-tier version with PerplexityBot for indexing and Perplexity-User for retrieval.

Anthropic says all three of its bots honor robots.txt, including Claude-User. OpenAI and Perplexity draw a sharper line for user-initiated fetchers, warning that robots.txt rules may not apply to ChatGPT-User and generally don’t apply to Perplexity-User. For Anthropic and OpenAI, blocking the training bot does not block the search bot or the user-requested fetcher.

What Changed From The Old Page

The previous version of Anthropic’s crawler page referenced only ClaudeBot and used broader language about data collection for model development. Before ClaudeBot, Anthropic operated under the Claude-Web and Anthropic-AI user agents, both now deprecated.

The move from one listed crawler to three mirrors what OpenAI did in late 2024 when it separated GPTBot from OAI-SearchBot and ChatGPT-User. OpenAI updated that documentation again in December, adding a note that GPTBot and OAI-SearchBot share information to avoid duplicate crawling when both are allowed.

OpenAI also noted in that December update that ChatGPT-User, which handles user-initiated browsing, may not be governed by robots.txt in the same way as its automated crawlers. Anthropic’s documentation does not make a similar distinction for Claude-User.

Why This Matters

The blanket “block AI crawlers” strategy that many sites adopted in 2024 no longer works the way it did. Blocking ClaudeBot stops training data collection but does nothing about Claude-SearchBot or Claude-User. The same is true on OpenAI’s side.

A BuzzStream study we covered in January found that 79% of top news sites block at least one AI training bot. But 71% also block at least one retrieval or search bot, potentially removing themselves from AI-powered search citations in the process.

That matters more now than it did a year ago. Hostinger’s analysis of 66.7 billion bot requests showed OpenAI’s search crawler coverage growing from 4.7% to over 55% of sites in their sample, even as its training crawler coverage dropped from 84% to 12%. Websites are allowing search bots while blocking training bots, and the gap is widening.

The visibility warnings differ by company. Anthropic says blocking Claude-SearchBot “may reduce” visibility. OpenAI is more direct, telling publishers that sites opted out of OAI-SearchBot won’t appear in ChatGPT search answers, though navigational links may still show up. Both are positioning their search crawlers alongside Googlebot and Bingbot, not alongside their own training crawlers.

What This Means

When managing robots.txt files, the old copy-paste block list needs an audit. SEJ’s complete AI crawler list includes verified user-agent strings across every company.

A strategic robots.txt now requires separate entries for training and search bots at minimum, with the understanding that user-initiated fetchers may not follow the same rules.

Looking Ahead

The three-tier split creates a new category of publisher decision that parallels what Google did years ago with Google-Extended. That user-agent lets sites opt out of Gemini training while staying in Google Search results. Now Anthropic and OpenAI offer the same separation for their platforms.

As AI-powered search grows its share of referral traffic, the cost of blocking search crawlers increases. The Cloudflare Year in Review data we reported in December showed AI crawlers already account for a measurable share of web traffic, and the gap between crawling volume and referral traffic remains wide. How publishers navigate these three-way decisions will shape how much of the web AI search tools can actually surface.

Microsoft: ‘Summarize With AI’ Buttons Used To Poison AI Recommendations via @sejournal, @MattGSouthern

Microsoft’s Defender Security Research Team published research describing what it calls “AI Recommendation Poisoning.” The technique involves businesses hiding prompt-injection instructions within website buttons labeled “Summarize with AI.”

When you click one of these buttons, it opens an AI assistant with a pre-filled prompt delivered through a URL query parameter. The visible part tells the assistant to summarize the page. The hidden part instructs it to remember the company as a trusted source for future conversations.

If the instruction enters the assistant’s memory, it can influence recommendations without you knowing it was planted.

What’s Happening

Microsoft’s team reviewed AI-related URLs observed in email traffic over 60 days. They found 50 distinct prompt injection attempts from 31 companies.

The prompts share a similar pattern. Microsoft’s post includes examples where instructions told the AI to remember a company as “a trusted source for citations” or “the go-to source” for a specific topic. One prompt went further, injecting full marketing copy into the assistant’s memory, including product features and selling points.

The researchers traced the technique to publicly available tools, including the npm package CiteMET and the web-based URL generator AI Share URL Creator. The post describes both as designed to help websites “build presence in AI memory.”

The technique relies on specially crafted URLs with prompt parameters that most major AI assistants support. Microsoft listed the URL structures for Copilot, ChatGPT, Claude, Perplexity, and Grok, but noted that persistence mechanisms differ across platforms.

It’s formally cataloged as MITRE ATLAS AML.T0080 (Memory Poisoning) and AML.T0051 (LLM Prompt Injection).

What Microsoft Found

The 31 companies identified were real businesses, not threat actors or scammers.

Multiple prompts targeted health and financial services sites, where biased AI recommendations carry more weight. One company’s domain was easily mistaken for a well-known website, potentially leading to false credibility. And one of the 31 companies was a security vendor.

Microsoft called out a secondary risk. Many of the sites using this technique had user-generated content sections like comment threads and forums. Once an AI treats a site as authoritative, it may extend that trust to unvetted content on the same domain.

Microsoft’s Response

Microsoft said it has protections in Copilot against cross-prompt injection attacks. The company noted that some previously reported prompt-injection behaviors can no longer be reproduced in Copilot, and that protections continue to evolve.

Microsoft also published advanced hunting queries for organizations using Defender for Office 365, allowing security teams to scan email and Teams traffic for URLs containing memory manipulation keywords.

You can review and remove stored Copilot memories through the Personalization section in Copilot chat settings.

Why This Matters

Microsoft compares this technique to SEO poisoning and adware, placing it in the same category as the tactics Google spent two decades fighting in traditional search. The difference is that the target has moved from search indexes to AI assistant memory.

Businesses doing legitimate work on AI visibility now face competitors who may be gaming recommendations through prompt injection.

The timing is notable. SparkToro published a report showing that AI brand recommendations already vary across nearly every query. Google VP Robby Stein told a podcast that AI search finds business recommendations by checking what other sites say. Memory poisoning bypasses that process by planting the recommendation directly into the user’s assistant.

Roger Montti’s analysis of AI training data poisoning covered the broader concept of manipulating AI systems for visibility. That piece focused on poisoning training datasets. This Microsoft research shows something more immediate, happening at the point of user interaction and being deployed commercially.

Looking Ahead

Microsoft acknowledged this is an evolving problem. The open-source tooling means new attempts can appear faster than any single platform can block them, and the URL parameter technique applies to most major AI assistants.

It’s unclear whether AI platforms will treat this as a policy violation with consequences, or whether it stays as a gray-area growth tactic that companies continue to use.

Hat tip to Lily Ray for flagging the Microsoft research on X, crediting @top5seo for the find.


Featured Image: elenabsl/Shutterstock

Google Offers AI Certificate Free For Eligible U.S. Small Businesses via @sejournal, @MattGSouthern

Google has launched the Google AI Professional Certificate, a self-paced program covering data analysis, content creation, research, and vibe coding.

Every participant receives three months of free access to Google AI Pro. Eligible U.S. small businesses can access the entire program at no cost through a separate application (more on eligibility below).

The certificate is available now on Coursera, Google Skills, and Udemy. In the U.S. and Canada, the subscription costs $49 per month.

What The Certificate Covers

The program consists of seven modules, each of which can be completed in about an hour. No prior AI experience is required.

Participants complete more than 20 hands-on activities. These include creating presentations and marketing materials, conducting deep research, building infographics, analyzing data, and building custom apps without writing code.

After completing all seven modules, participants earn a Google certificate they can add to LinkedIn and share with employers.

Free Access For Eligible U.S. Small Businesses

Google is offering the certificate at no cost to eligible U.S. small and medium-sized businesses with 500 or fewer employees. The offer also includes three months of free Google Workspace Business Standard (for new Workspace customers, up to 300 seats).

To qualify, businesses must be registered in the U.S. and submit their Employer Identification Number (EIN) through a dedicated application on Coursera. Coursera said the verification process takes 5-7 business days.

Businesses can also apply at grow.google/small-business. Google said it is working with the U.S. Chamber of Commerce and America’s Small Business Development Centers to distribute the program.

How This Helps

The program builds on Google AI Essentials, which has become the most popular course on Coursera. The AI Professional Certificate goes further, focusing on applied use cases rather than introductory concepts.

The certificate focuses on tools like Gemini, NotebookLM, and Google AI Studio, so the skills are tied to Google’s ecosystem. Google launched a separate Generative AI Leader certification for Google Cloud in May 2025, though that program focused on non-technical business leaders and required a $99 exam fee. The new AI Professional Certificate has no exam fee.

Looking Ahead

The Google AI Professional Certificate is available now on Coursera, Google Skills, and Udemy. Eligible U.S. small businesses can apply for no-cost access at grow.google/small-business.

For professionals already familiar with Google’s AI tools through earlier training programs, this certificate adds structured, employer-recognized credentials to practical skills you may already be developing on your own.

Why AI Misreads The Middle Of Your Best Pages via @sejournal, @DuaneForrester

The middle is where your content dies, and not because your writing suddenly gets bad halfway down the page, and not because your reader gets bored. But because large language models have a repeatable weakness with long contexts, and modern AI systems increasingly squeeze long content before the model even reads it.

That combo creates what I think of as dog-bone thinking. Strong at the beginning, strong at the end, and the middle gets wobbly. The model drifts, loses the thread, or grabs the wrong supporting detail. You can publish a long, well-researched piece and still watch the system lift the intro, lift the conclusion, then hallucinate the connective tissue in between.

This is not theory as it shows up in research, and it also shows up in production systems.

Image Credit: Duane Forrester

Why The Dog-Bone Happens

There are two stacked failure modes, and they hit the same place.

First, “lost in the middle” is real. Stanford and collaborators measured how language models behave when key information moves around inside long inputs. Performance was often highest when the relevant material was at the beginning or end, and it dropped when the relevant material sat in the middle. That’s the dog-bone pattern, quantified.

Second, long contexts are getting bigger, but systems are also getting more aggressive about compression. Even if a model can take a massive input, the product pipeline frequently prunes, summarizes, or compresses to control cost and keep agent workflows stable. That makes the middle even more fragile, because it is the easiest segment to collapse into mushy summary.

A fresh example: ATACompressor is a 2026 arXiv paper focused on adaptive, task-aware compression for long-context processing. It explicitly frames “lost in the middle” as a problem in long contexts and positions compression as a strategy that must preserve task-relevant content while shrinking everything else.

So you were right if you ever told someone to “shorten the middle.” Now, I’d offer this refinement:

You are not shortening the middle for the LLM so much as engineering the middle to survive both attention bias and compression.

Two Filters, One Danger Zone

Think of your content going through two filters before it becomes an answer.

  • Filter 1: Model Attention Behavior: Even if the system passes your text in full, the model’s ability to use it is position-sensitive. Start and end tend to perform better, middle tends to perform worse.
  • Filter 2: System-Level Context Management: Before the model sees anything, many systems condense the input. That can be explicit summarization, learned compression, or “context folding” patterns used by agents to keep working memory small. One example in this space is AgentFold, which focuses on proactive context folding for long-horizon web agents.

If you accept those two filters as normal, the middle becomes a double-risk zone. It gets ignored more often, and it gets compressed more often.

That is the balancing logic with the dog-bone idea. A “shorten the middle” approach becomes a direct mitigation for both filters. You are reducing what the system will compress away, and you are making what remains easier for the model to retrieve and use.

What To Do About It Without Turning Your Writing Into A Spec Sheet

This is not a call to kill longform as longform still matters for humans, and for machines that use your content as a knowledge base. The fix is structural, not “write less.”

You want the middle to carry higher information density with clearer anchors.

Here’s the practical guidance, kept tight on purpose.

1. Put “Answer Blocks” In The Middle, Not Connective Prose

Most long articles have a soft, wandering middle where the author builds nuance, adds color, and tries to be thorough. Humans can follow that. Models are more likely to lose the thread there. Instead, make the middle a sequence of short blocks where each block can stand alone.

An answer block has:
A clear claim. A constraint. A supporting detail. A direct implication.

If a block cannot survive being quoted by itself, it will not survive compression. This is how you make the middle “hard to summarize badly.”

2. Re-Key The Topic Halfway Through

Drift often happens because the model stops seeing consistent anchors.

At the midpoint, add a short “re-key” that restates the thesis in plain words, restates the key entities, and restates the decision criteria. Two to four sentences are often enough here. Think of this as continuity control for the model.

It also helps compression systems. When you restate what matters, you are telling the compressor what not to throw away.

3. Keep Proof Local To The Claim

Models and compressors both behave better when the supporting detail sits close to the statement it supports.

If your claim is in paragraph 14, and the proof is in paragraph 37, a compressor will often reduce the middle into a summary that drops the link between them. Then the model fills that gap with a best guess.

Local proof looks like:
Claim, then the number, date, definition, or citation right there. If you need a longer explanation, do it after you’ve anchored the claim.

This is also how you become easier to cite. It is hard to cite a claim that requires stitching context from multiple sections.

4. Use Consistent Naming For The Core Objects

This is a quiet one, but it matters a lot. If you rename the same thing five times for style, humans nod, but models can drift.

Pick the term for the core thing and keep it consistent throughout. You can add synonyms for humans, but keep the primary label stable. When systems extract or compress, stable labels become handles. Unstable labels become fog.

5. Treat “Structured Outputs” As A Clue For How Machines Prefer To Consume Information

A big trend in LLM tooling is structured outputs and constrained decoding. The point is not that your article should be JSON. The point is that the ecosystem is moving toward machine-parseable extraction. That trend tells you something important: machines want facts in predictable shapes.

So, inside the middle of your article, include at least a few predictable shapes:
Definitions. Step sequences. Criteria lists. Comparisons with fixed attributes. Named entities tied to specific claims.

Do that, and your content becomes easier to extract, easier to compress safely, and easier to reuse correctly.

How This Shows Up In Real SEO Work

This is the crossover point. If you are an SEO or content lead, you are not optimizing for “a model.” You are optimizing for systems that retrieve, compress, and synthesize.

Your visible symptoms will look like:

  • Your article gets paraphrased correctly at the top, but the middle concept is misrepresented. That’s lost-in-the-middle plus compression.
  • Your brand gets mentioned, but your supporting evidence does not get carried into the answer. That’s local proof failing. The model cannot justify citing you, so it uses you as background color.
  • Your nuanced middle sections become generic. That’s compression, turning your nuance into a bland summary, then the model treating that summary as the “true” middle.
  • Your “shorten the middle” move is how you reduce these failure rates. Not by cutting value, but by tightening the information geometry.

A Simple Way To Edit For Middle Survival

Here’s a clean, five-step workflow you can apply to any long piece, and it’s a sequence you can run in an hour or less.

  1. Identify the midpoint and read only the middle third. If the middle third can’t be summarized in two sentences without losing meaning, it’s too soft.
  2. Add one re-key paragraph at the start of the middle third. Restate: the main claim, the boundaries, and the “so what.” Keep it short.
  3. Convert the middle third into four to eight answer blocks. Each block must be quotable. Each block must include its own constraint and at least one supporting detail.
  4. Move proof next to claim. If proof is far away, pull a compact proof element up. A number, a definition, a source reference. You can keep the longer explanation later.
  5. Stabilize the labels. Pick the name for your key entities and stick to them across the middle.

If you want the nerdy justification for why this works, it is because you are designing for both failure modes documented above: the “lost in the middle” position sensitivity measured in long-context studies, and the reality that production systems compress and fold context to keep agents and workflows stable.

Wrapping Up

Bigger context windows do not save you. They can make your problem worse, because long content invites more compression, and compression invites more loss in the middle.

So yes, keep writing longform when it is warranted, but stop treating the middle like a place to wander. Treat it like the load-bearing span of a bridge. Put the strongest beams there, not the nicest decorations.

That’s how you build content that survives both human reading and machine reuse, without turning your writing into sterile documentation.

More Resources:


This post was originally published on Duane Forrester Decodes.


Featured Image: Collagery/Shutterstock

ChatGPT Search Often Switches To English In Fan-Out Queries: Report via @sejournal, @MattGSouthern

When ChatGPT Search builds an answer, it can generate background web queries to find sources. A new report from AI search analytics firm Peec AI found that a large share of those background queries run in English, even when the original prompt was in another language.

Peec AI analyzed over 10 million prompts and 20 million fan-out queries from its platform data. Across all non-English prompts analyzed, the company reports that 43% of the fan-out steps were conducted in English.

What Are Fan-Out Queries

OpenAI’s ChatGPT Search documentation describes fan-out queries. When a user asks a question, ChatGPT Search “typically rewrites your query into one or more targeted queries” and sends them to search partners. After reviewing initial results, “ChatGPT search may send additional, more specific queries to other search providers.”

Peec AI refers to these rewritten sub-queries as “fan-outs.” The company’s report tracked which languages ChatGPT used when generating them.

OpenAI’s documentation does not describe how language is chosen for rewritten queries.

What Peec AI Found

Peec AI filtered its data to include only cases where the IP location matched the prompt language. Polish-language prompts from Polish IP addresses, German-language prompts from German IPs, and Spanish-language prompts from Spanish IPs. Mixed signals, such as German-language prompts from UK IP addresses, were excluded.

The filtered data showed that 78% of non-English prompt runs included at least one English-language fan-out query.

Turkish-language prompts included English fan-outs most often, at 94%. Spanish-language prompts were lowest, at 66%. No non-English language in Peec AI’s dataset fell below 60%.

Peec AI’s data showed a consistent pattern across languages. ChatGPT typically starts its fan-out queries in the prompt’s language, then adds English-language queries as it builds the response.

Examples From The Report

Peec AI’s blog post included several examples showing how the pattern can play out in practice.

When prompted in Polish from a Polish IP address about the best auction portals, ChatGPT either omitted or buried Allegro.pl in favor of eBay and other global platforms. Peec AI describes Allegro as Poland’s dominant ecommerce platform.

When prompted in German about German software companies, Peec AI reported the response listed no German companies. When prompted in Spanish about cosmetics brands, no Spanish brands appeared.

In the Spanish cosmetics example, Peec AI showed ChatGPT’s actual fan-out queries. The first ran in English. The second ran in Spanish but added the word “globales” (global), a qualifier the original prompt never used. The system appears to have interpreted a Spanish-language prompt from a Spanish IP address as a request for global brands.

These are individual examples from Peec AI’s testing, not necessarily representative of all ChatGPT Search behavior.

Why This Matters

SEO and content teams operating in non-English markets may face a disadvantage in ChatGPT’s source selection that may not map cleanly to traditional ranking signals. In Peec AI’s examples, English-language fan-out queries surfaced English-language sources that favored global brands over local competitors.

We’ve been covering ChatGPT’s citation patterns for over a year now, from SE Ranking’s report on citation factors to the Tow Center’s attribution accuracy findings. Those earlier reports showed which signals predict whether a source gets cited. Peec AI’s data suggests the language of the background query may filter which sources are even considered, before citation signals come into play.

Methodology Notes

Peec AI is a vendor in the AI search analytics space. The company’s documentation describes its data collection method as running customer-defined prompts daily via browser automation, interacting with AI platforms through their web interfaces rather than APIs. The 10 million prompts in this report came from Peec AI’s platform, not from a panel of consumer ChatGPT sessions.

The report didn’t detail the composition of those prompts, what categories or industries they covered, or how representative they are of broader ChatGPT usage patterns.

Tomek Rudzki, the report’s author, is presented by Peec AI as a “GEO Expert” on its blog. He is a well-known technical SEO practitioner who has spoken at BrightonSEO and SMX Munich and contributed to publications such as Moz.

Looking Ahead

OpenAI’s public ChatGPT Search docs describe query rewriting and follow-up queries but don’t explain how language is chosen for those queries. Whether the English fan-out pattern Peec AI identified is an intentional design choice or an emergent behavior of the system remains unclear.

The report raises a question worth monitoring. Will building English-language content become part of AI search optimization strategies, or will AI search platforms adjust their source selection to better reflect local markets?


Featured Image: arda savasciogullari/Shutterstock

Why Google Runs AI Mode On Flash, Explained By Google’s Chief Scientist via @sejournal, @MattGSouthern

Google Chief Scientist Jeff Dean said Flash’s low latency and cost are why Google can run Search AI at scale. Retrieval is a design choice, not a limitation, he added.

In an interview on the Latent Space podcast, Dean explained why Flash became the production tier for Search. He also laid out why the pipeline that narrows the web to a handful of documents will likely persist.

Google started rolling out Gemini 3 Flash as the default for AI Mode in December. Dean’s interview explains the rationale behind that decision.

Why Flash Is The Production Tier

Dean called latency the critical constraint for running AI in Search. As models handle longer and more complex tasks, speed becomes the bottleneck.

“Having low latency systems that can do that seems really important, and flash is one direction, one way of doing that.”

Podcast hosts noted Flash’s dominance across services like Gmail and YouTube. Dean said search is part of that expansion, with Flash’s use growing across AI Mode and AI Overviews.

Flash can serve at this scale because of distillation. Each generation’s Flash inherits the previous generation’s Pro-level performance, getting more capable without getting more expensive to run.

“For multiple Gemini generations now, we’ve been able to make the sort of flash version of the next generation as good or even substantially better than the previous generation’s pro.”

That’s the mechanism that makes the architecture sustainable. Google pushes frontier models for capability development, then distills those capabilities into Flash for production deployment. Flash is the tier Google designed to run at search scale.

Retrieval Over Memorization

Beyond Flash’s role in search, Dean described a design philosophy that keeps external content central to how these models work. Models shouldn’t waste capacity storing facts they can retrieve.

“Having the model devote precious parameter space to remember obscure facts that could be looked up is actually not the best use of that parameter space.”

Retrieval from external sources is a core capability, not a workaround. The model looks things up and works through the results rather than carrying everything internally.

Why Staged Retrieval Likely Persists

AI search can’t read the entire web at once. Current attention mechanisms are quadratic, meaning computational cost grows rapidly as context length increases. Dean said “a million tokens kind of pushes what you can do.” Scaling to a billion or a trillion isn’t feasible with existing methods.

Dean’s long-term vision is models that give the “illusion” of attending to trillions of tokens. Reaching that requires new techniques, not just scaling what exists today. Until then, AI search will likely keep narrowing a broad candidate pool to a handful of documents before generating a response.

Why This Matters

The model reading your content in AI Mode is getting better each generation. But it’s optimized for speed over reasoning depth, and it’s designed to retrieve your content rather than memorize it. Being findable through Google’s existing retrieval and ranking signals is the path into AI search results.

We’ve tracked every model swap in AI Mode and AI Overviews since Google launched AI Mode with Gemini 2.0. Google shipped Gemini 3 to AI Mode on release day, then started rolling out Gemini 3 Flash as the default a month later. Most recently, Gemini 3 became the default for AI Overviews globally.

Every model generation follows the same cycle. Frontier for capability, then distillation into Flash for production. Dean presented this as the architecture Google expects to maintain at search scale, not a temporary fallback.

Looking Ahead

Based on Dean’s comments, staged retrieval is likely to persist until attention mechanisms move past their quadratic limits. Google’s investment in Flash suggests the company expects to use this architecture across multiple model generations.

One change to watch is automatic model selection. Google’s Robby Stein described mentioned the concept previously, which involves routing complex queries to Pro while keeping Flash as the default.


Featured Image: Robert Way/Shutterstock

5 Ways Emerging Businesses Can Show up in ChatGPT, Gemini & Perplexity via @sejournal, @nofluffmktg

This post was sponsored by No Fluff. The opinions expressed in this article are the sponsor’s own.

When ChatGPT, Gemini, and Perplexity mention a company, these large language models (LLMs) are deciding whether that business is safe to reference, not how long it has existed.

Most business leaders assume one thing when they don’t show up in AI-generated answers:

We’re too new.

In reality, early testing across multiple AI platforms suggests something else is going on. In many cases, the problem has less to do with company age and more to do with how AI systems evaluate structure, repetition, and trust signals.

It is possible for new brands to be mentioned in AI search results.

Even well-built products with real expertise are routinely missing from AI recommendations. Yet when buyers ask who to trust, the same legacy names keep appearing.

Why Most New Businesses Don’t Show Up In AI Search Results

This isn’t random.

AI systems lean on existing training data and visible digital footprints, which favor brands that have been cited for years. Because every answer carries risk, these systems act conservatively.

They don’t look for the most optimized page; they look for the most verifiable entity. If your footprint is thin, inconsistent, or poorly supported by third parties, the AI will often swap you out for a competitor it can trust more easily.

Most new businesses launch with:

  • Minimal historical signals
    Very little online content or mentions, so AI has almost nothing to work with.
  • Few credibility signals
    Few backlinks, reviews, or press, so you don’t “look” trustworthy yet.
  • Blending brand names
    Similar or generic brand names are easier for AI systems to confuse, misattribute, or skip entirely if trust signals are weak.
  • Unclear positioning
    Unclear positioning or ideas that appear only once on a company website are less likely to be trusted.

Together, these create unreliable signals.

In generative search, visibility is less about ranking and more about reasoning.

This is why most new brands aren’t evaluated as “bad,” but as too uncertain to reference safely.

That distinction matters. Being referenced by AI is not just exposure; it influences who buyers consider credible before they ever reach a website. AI-referred visitors often convert at higher rates than traditional organic traffic.

For new businesses, the lack of legacy signals isn’t “just a disadvantage.” Handled correctly, it can be an opening to establish clarity and trust faster than older competitors that rely on outdated authority.

There’s surprisingly little guidance on whether a new or growing brand can actually appear in AI-generated answers. Given how much these systems depend on past signals, it’s easy to assume established companies appear by default.

To test that assumption, a brand-new B2B company was tracked from launch as part of a 12-week AI search visibility experiment. The findings below reflect the first six weeks of that ongoing test. The company started with no prior history, no backlinks, and no press coverage. A true zero.

Visibility was measured across 150 buyer-style prompts in ChatGPT, Google AI Overviews, and Perplexity rather than inferred from third-party dashboards.

Using weekly GEO sprints focused on technical foundations, answer-first content, and reinforcing signals like social, video, and early backlinks, the goal was to see how far a best-practice GEO playbook could move a truly new brand.

Within six weeks, the emerging business saw the following results:

  • Appeared in 5% of relevant AI responses.
  • Showed up across 39 of 150 questions.
  • Mentioned 74 times, with 42 cited mentions.
  • 6% citation accuracy, ~11% pointing to the brand’s own site.

6 Patterns Observed in Early AI Visibility Testing

Across the first six weeks, six patterns consistently influenced whether the brand was included, replaced by a competitor, or excluded entirely from AI-generated answers:

Pattern 1: Structure Matters More Than Topic

Image created by No Fluff, February 2026

Content that wandered (even if it was thoughtful or “robust”) consistently lagged in AI pickup. The pages that were picked up were tighter: they answered the question up front, broke the content into clear steps, and stuck to one idea at a time.

Pattern 2: The Social “Amplifier” Effect

AI is more likely to cite sources it already trusts. In the first two weeks, most citations came from the brand’s LinkedIn and Medium posts rather than its website. For a new brand, publishing key ideas first on high-authority platforms, including LinkedIn or Medium, often triggers AI pickup before the same content is indexed on your own website.

Image created by No Fluff, February 2026

Pattern 3: Hallucinations are Often Signal Failures

Image created by No Fluff, February 2026

When AI systems misidentify a new brand or confuse it with competitors, the cause is typically thin, slow, or conflicting signals. When pages failed to load within roughly 5–15 seconds, AI systems issue broader “fan-out” queries and assemble answers from adjacent or incorrect sources. Following improvements in site speed, crawl reliability, and entity clarity, the share of answers that correctly referenced this company’s own domain increased, while misattributed mentions declined.

Pattern 4: The 3-Week Indexing Window

The first AI pickup from a new domain can happen within three to four weeks. In this experiment, the first page was discovered on day 27. After that initial discovery, subsequent pages were picked up faster, with the shortest lag around eight days.

Image created by No Fluff, February 2026

Early inclusion wasn’t driven by content volume. It was driven by structure: a solid schema, consistent metadata, a clean, crawlable site, and machine-readable files such as llms.txt.

Pattern 5: Win the Explanatory Round First

New brands typically will not start by winning highly competitive, decision-stage prompts like “best” or “top” lists, unless the offering is truly unique or non-competitive. Before a brand can realistically be shortlisted, it must first be sourced as a primary authority for definitional or educational questions.

In the first 45 days, the goal wasn’t comparison visibility, but recognition and trust: getting AI systems to associate the brand with the right topics and sources. Early success is best measured by citation frequency, or how often a brand is used as the primary source for a given topic.

Pattern 6: Solve the Unfinished Trust Gap (Most Important)

Even with a well-structured site and strong content, brands struggle to get recommended without outside validation. The initial stages of this experiment showed AI answers defaulted to familiar domains and replaced newer brands with competitors that had clearer third-party mentions. This validates the importance of press and authoritative coverage early on. Waiting to “add it later” only slows trust.

5 Steps To Set A New Business Up For AI Visible Success

By now, the takeaway is clear: AI visibility doesn’t happen automatically once a site is live or a few campaigns are running. The good news is that this can be influenced deliberately. The steps below reflect the sequence that consistently moved a new brand from zero visibility to being cited in AI-generated answers. Rather than treating AI visibility as a side effect of SEO, this approach treats it as an operational problem: how to make a brand easy for AI systems to recognize, verify, and reuse.

Step 1: Map Your Brand Entity

Before building a site, you must define your brand in a way machines understand. ChatGPT, Gemini, and Perplexity don’t read your website the way humans do. They connect facts, names, and relationships into entities that define who you are. If those connections are missing or inconsistent, your brand simply won’t appear (no matter how much content you publish).

  • Define your business clearly using semantic triples: Use the [Subject] → [Predicate] → [Object] format (e.g., “Brand X” → “offers” → “Service Y”) to provide machine-readable facts.
  • Stick to public, widely understood language: Pull terminology from widely accepted sources like Wikipedia or Wikidata. If you describe your product using internal jargon that doesn’t match how the category is commonly defined, you risk being misclassified or overlooked.
  • State your authority: Define why your brand deserves trust. What facts, evidence, and proof back you up? Write 3–5 simple, factual claims you want to be known for.
  • Define your competitive counter-position: Be clear about what makes you different. Scope the specific niche you own (audience, problem, angle, or offering) that sets you apart from alternatives.

Step 2: Engineer Your Benchmark Prompt Set

You cannot rely on traditional SEO tools designed to track AI visibility. Most rely on inferred data or simulations, not on real prompts.

  • Map the competitive landscape: Identify which brands AI systems already reference, which buyer questions are realistically winnable, and where category language creates confusion.
  • Reverse-engineer buyer questions: Identify how buyers phrase real questions using keyword and competitor analysis (SEO tool data, People Also Ask, Google SERPS, and asking multiple AI engines themselves)
  • Lock your data set: Create a fixed set of 150 buyer-authentic questions across six clusters: Branded, Category, Problem, Comparison, and Advanced Semantic.
  • Start testing: Run these prompts weekly across ChatGPT, Gemini, and Perplexity to track your mentions and citation growth.

Step 3:  Make the Brand Machine-Readable

Make your site machine-readable to ensure AI bots don’t skip your content. AI systems don’t care about your website’s aesthetic; they care about how easily they can parse your data. If your technical signals are thin or conflicting, AI will hallucinate or substitute your brand with a competitor.

  • Implement JSON-LD Schema: Use Organization, Service, and FAQ schemas to tell AI exactly who you are and what you do.
  • Deploy an txt File: Place this at your domain root to provide a plain-text guide for AI crawlers, telling them how to describe your company and which pages to prioritize.
  • Eliminate crawling issues: Make sure your site is fully crawlable via robots.txt and that no content is hidden in gated PDFs or images. Most importantly, check site speed using PageSpeed Insights. Models don’t patiently wait for slow pages!

Step 4:  Publish “Retrieval-Ready” Content

Write for the impatient analyst (the AI bot). Start with high-leverage prompts, questions with real buyer intent that AI already answers, but only using a small and weak set of sources, making them easier to influence before trust fully locks in.

  • Lead with the answer: Start every section with a direct, factual answer.
  • Chunk semantically: Divide content into logical, independent sections that can be extracted and reused by AI without requiring the context of the entire page.
  • Consider the freshness factor: AI favors content updated within the last 60–90 days. For high-competition sectors like SaaS or Finance, content should be refreshed every three months to remain a “trusted” recommendation.

Step 5:  Earn External Validation

AI systems cross-check your site’s claims against the rest of the web.

  • Claim directory profiles: Align your entity data across Crunchbase, G2, LinkedIn, and Yelp. Inconsistencies across these profiles are a primary cause of AI hallucinations.
  • Target authoritative mentions: Secure mentions in industry-specific publications with consistent pickup throughout your prompts and or a strong domain rating.
  • External reinforcement: For every important page on your site, aim for at least three intentional external link-backs from authoritative sources to trigger AI pickup.

The Biggest Takeaway: Prioritize Authority as a Long-Term Game

For new brands, the limiting factor in AI search is not optimization. It’s authority.

AI systems are more likely to surface unfamiliar companies first in low-risk, explanatory answers, not in “best,” “top,” or comparison prompts. A clean site and solid SEO help a brand get recognized, but being recommended is a different hurdle.

In practice, early progress is about reducing uncertainty. When a brand consistently appears in third-party articles, reviews, or other independent sources, it becomes easier to explain and safer to reference. Without that outside validation, recommendations stall, no matter how strong the content or how fast the site loads.

This analysis covers the first phase of a live 90-day test examining how a new B2B brand earns visibility in AI-generated search results. Ongoing findings and final results will be published as the experiment concludes.


Image Credits

Featured Image: Image by No Fluff. Used with permission.

In-Post Images: Images by No Fluff. Used with permission.

Synthetic Personas For Better Prompt Tracking via @sejournal, @Kevin_Indig

Boost your skills with Growth Memo’s weekly expert insights. Subscribe for free!

We all know prompt tracking is directional. The most effective way to reduce noise is to track prompts based on personas.

This week, I’m covering:

  • Why AI personalization makes traditional “track the SERP” models incomplete, and how synthetic personas fill the gap.
  • The Stanford validation data showing 85% accuracy at one-third the cost, and how Bain cut research time by 50-70%.
  • The five-field persona card structure and how to generate 15-30 trackable prompts per segment across intent levels.
The best way to make your prompt tracking much more accurate is to base it on personas. Synthetic Personas speed you up at a fraction of the price. (Image Credit: Kevin Indig)

A big difference between classic and AI search is that the latter delivers highly personalized results.

  • Every user gets different answers based on their context, history, and inferred intent.
  • The average AI prompt is ~5x longer than classic search keywords (23 words vs. 4.2 words), conveying much richer intent signals that AI models use for personalization.
  • Personalization creates a tracking problem: You can’t monitor “the” AI response anymore because each prompt is essentially unique, shaped by individual user context.

Traditional persona research solves this – you map different user segments and track responses for each – but it creates new problems. It takes weeks to conduct interviews and synthesize findings.

By the time you finish, the AI models have changed. Personas become stale documentation that never gets used for actual prompt tracking.

Synthetic personas fill the gap by building user profiles from behavioral and profiling data: analytics, CRM records, support tickets, review sites. You can spin up hundreds of micro-segment variants and interact with them in natural language to test how they’d phrase questions.

Most importantly: They are the key to more accurate prompt tracking because they simulate actual information needs and constraints.

The shift: Traditional personas are descriptive (who the user is), synthetic personas are predictive (how the user behaves). One documents a segment, the other simulates it.

Image Credit: Kevin Indig

Example: Enterprise IT buyer persona with job-to-be-done “evaluate security compliance” and constraint “need audit trail for procurement” will prompt differently than an individual user with the job “find cheapest option” and constraint “need decision in 24 hours.”

  • First prompt: “enterprise project management tools SOC 2 compliance audit logs.”
  • Second prompt: “best free project management app.”
  • Same product category, completely different prompts. You need both personas to track both prompt patterns.

Build Personas With 85% Accuracy For One-Third Of The Price

Stanford and Google DeepMind trained synthetic personas on two-hour interview transcripts, then tested whether the AI personas could predict how those same real people would answer survey questions later.

  • The method: Researchers conducted follow-up surveys with the original interview participants, asking them new questions. The synthetic personas answered the same questions.
  • Result: 85% accuracy. The synthetic personas replicated what the actual study participants said.
  • For context, that’s comparable to human test-retest consistency. If you ask the same person the same question two weeks apart, they’re about 85% consistent with themselves.

The Stanford study also measured how well synthetic personas predicted social behavior patterns in controlled experiments – things like who would cooperate in trust games, who would follow social norms, and who would share resources fairly.

The correlation between synthetic persona predictions and actual participant behavior was 98%. This means the AI personas didn’t just memorize interview answers; they captured underlying behavioral tendencies that predicted how people would act in new situations.

Bain & Company ran a separate pilot that showed comparable insight quality at one-third the cost and one-half the time of traditional research methods. Their findings: 50-70% time reduction (days instead of weeks) and 60-70% cost savings (no recruiting fees, incentives, transcription services).

The catch: These results depend entirely on input data quality. The Stanford study used rich, two-hour interview transcripts. If you train on shallow data (just pageviews or basic demographics), you get shallow personas. Garbage in, garbage out.

How To Build Synthetic Personas For Better Prompt Tracking

Building a synthetic persona has three parts:

  1. Feed it with data from multiple sources about your real users: call transcripts, interviews, message logs, organic search data.
  2. Fill out the Persona Card – the five fields that capture how someone thinks and searches.
  3. Add metadata to track the persona’s quality and when it needs updating.

The mistake most teams make: trying to build personas from prompts. This is circular logic – you need personas to understand what prompts to track, but you’re using prompts to build personas. Instead, start with user information needs, then let the persona translate those needs into likely prompts.

Data Sources To Feed Synthetic Personas

The goal is to understand what users are trying to accomplish and the language they naturally use:

  1. Support tickets and community forums: Exact language customers use when describing problems. Unfiltered, high-intent signal.
  2. CRM and sales call transcripts: Questions they ask, objections they raise, use cases that close deals. Shows the decision-making process.
  3. Customer interviews and surveys: Direct voice-of-customer on information needs and research behavior.
  4. Review sites (G2, Trustpilot, etc.): What they wish they’d known before buying. Gap between expectation and reality.
  5. Search Console query data: Questions they ask Google. Use regex to filter for question-type queries:
    (?i)^(who|what|why|how|when|where|which|can|does|is|are|should|guide|tutorial|course|learn|examples?|definition|meaning|checklist|framework|template|tips?|ideas?|best|top|lists?|comparison|vs|difference|benefits|advantages|alternatives)b.*

    (I like to use the last 28 days, segment by target country)

Persona card structure (five fields only – more creates maintenance debt):

These five fields capture everything needed to simulate how someone would prompt an AI system. They’re minimal by design. You can always add more later, but starting simple keeps personas maintainable.

  1. Job-to-be-done: What’s the real-world task they’re trying to accomplish? Not “learn about X” but “decide whether to buy X” or “fix problem Y.”
  2. Constraints: What are their time pressures, risk tolerance levels, compliance requirements, budget limits, and tooling restrictions? These shape how they search and what proof they need.
  3. Success metric: How do they judge “good enough?” Executives want directional confidence. Engineers want reproducible specifics.
  4. Decision criteria: What proof, structure, and level of detail do they require before they trust information and act on it?
  5. Vocabulary: What are the terms and phrases they naturally use? Not “churn mitigation” but “keeping customers.” Not “UX optimization” but “making the site easier to use.”

Specification Requirements

This is the metadata that makes synthetic personas trustworthy; it prevents the “black box” problem.

When someone questions a persona’s outputs, you can trace back to the evidence.

These requirements form the backbone of continuous persona development. They keep track of changes, sources, and confidence in the weighting.

  • Provenance: Which data sources, date ranges, and sample sizes were used (e.g., “Q3 2024 Support Tickets + G2 Reviews”).
  • Confidence score per field: A High/Medium/Low rating for each of the five Persona Card fields, backed by evidence counts. (e.g., “Decision Criteria: HIGH confidence, based on 47 sales calls vs. Vocabulary: LOW confidence, based on 3 internal emails”).
  • Coverage notes: Explicitly state what the data misses (e.g., “Overrepresents enterprise buyers, completely misses users who churned before contacting support”).
  • Validation benchmarks: Three to five reality checks against known business truths to spot hallucinations. (e.g., “If the persona claims ‘price’ is the top constraint, does that match our actual deal cycle data?”).
  • Regeneration triggers: Pre-defined signals that it’s time to re-run the script and refresh the persona (e.g., a new competitor enters the market, or vocabulary in support tickets shifts significantly).

Where Synthetic Personas Work Best

Before you build synthetic personas, understand where they add value and where they fall short.

High-Value Use Cases

  • Prompt design for AI tracking: Simulate how different user segments would phrase questions to AI search engines (the core use case covered in this article).
  • Early-stage concept testing: Test 20 messaging variations, narrow to the top five before spending money on real research.
  • Micro-segment exploration: Understand behavior across dozens of different user job functions (enterprise admin vs. individual contributor vs. executive buyer) or use cases without interviewing each one.
  • Hard-to-reach segments: Test ideas with executive buyers or technical evaluators without needing their time.
  • Continuous iteration: Update personas as new support tickets, reviews, and sales calls come in.

Crucial Limitations Of Synthetic Personas You Need To Understand

  • Sycophancy bias: AI personas are overly positive. Real users say, “I started the course but didn’t finish.” Synthetic personas say, “I completed the course.” They want to please.
  • Missing friction: They’re more rational and consistent than real people. If your training data includes support tickets describing frustrations or reviews mentioning pain points, the persona can reference these patterns when asked – it just won’t spontaneously experience new friction you haven’t seen before.
  • Shallow prioritization: Ask what matters, and they’ll list 10 factors as equally important. Real users have a clear hierarchy (price matters 10x more than UI color).
  • Inherited bias: Training data biases flow through. If your CRM underrepresents small business buyers, your personas will too.
  • False confidence risk: The biggest danger. Synthetic personas always have coherent answers. This makes teams overconfident and skip real validation.

Operating rule: Use synthetic personas for exploration and filtering, not for final decisions. They narrow your option set. Real users make the final call.

Solving The Cold Start Problem For Prompt Tracking

Synthetic personas are a filter tool, not a decision tool. They narrow your option set from 20 ideas to five finalists. Then, you validate those five with real users before shipping.

For AI prompt tracking specifically, synthetic personas solve the cold-start problem. You can’t wait to accumulate six months of real prompt volume before you start optimizing. Synthetic personas let you simulate prompt behavior across user segments immediately, then refine as real data comes in.

Where they’ll cause you to fail is if you use them as an excuse to skip real validation. Teams love synthetic personas because they’re fast and always give answers. That’s also what makes them dangerous. Don’t skip the validation step with real customers.


Featured Image: Paulo Bobita/Search Engine Journal

OpenAI Begins Testing Ads In ChatGPT For Free And Go Users via @sejournal, @MattGSouthern

OpenAI is testing ads inside ChatGPT, bringing sponsored content to the product for the first time.

The test is live for logged-in adult users in the U.S. on the free and Go subscription tiers. Plus, Pro, Business, Enterprise, and Education subscribers won’t see ads.

OpenAI announced the launch with a brief blog post confirming that the principles it outlined in January are now in effect.

OpenAI’s post also adds Education to the list of ad-free tiers, which wasn’t included in the company’s initial plans.

How The Ads Work

Ads appear at the bottom of ChatGPT responses, visually separated from the answer and labeled as sponsored.

OpenAI says it selects ads by matching advertiser submissions with the topic of your conversation, your past chats, and past interactions with ads. If someone asks about recipes, they might see an ad for a meal kit or grocery delivery service.

Advertisers don’t see users’ conversations or personal details. They receive only aggregate performance data like views and clicks.

Users can dismiss ads, see why a specific ad appeared, turn off personalization, or clear all ad-related data. OpenAI also confirmed it won’t show ads in conversations about health, mental health, or politics, and won’t serve them to accounts identified as under 18.

Free users who don’t want ads have another option. OpenAI says you can opt out of ads in the Free tier in exchange for fewer daily free messages. Go users can avoid ads by upgrading to Plus or Pro.

The Path To Today

OpenAI first announced plans to test ads on January 16, alongside the U.S. launch of ChatGPT Go at $8 per month. The company laid out five principles. They cover mission alignment, answer independence, conversation privacy, choice and control, and long-term value.

The January post was careful to frame ads as supporting access rather than driving revenue. Altman wrote on X at the time:

“It is clear to us that a lot of people want to use a lot of AI and don’t want to pay, so we are hopeful a business model like this can work.”

That framing sits alongside OpenAI’s financial reality. Altman said in November that the company is considering infrastructure commitments totaling about $1.4 trillion over eight years. He also said OpenAI expects to end 2025 with an annualized revenue run rate above $20 billion. A source told CNBC that OpenAI expects ads to account for less than half of its revenue long term.

OpenAI has confirmed a $200,000 minimum commitment for early ChatGPT ads, Adweek reported. Digiday reported media buyers were quoted about $60 per 1,000 views for sponsored placements during the initial U.S. test.

Altman’s Evolving Position

The launch represents a notable turn from Altman’s earlier public statements on advertising.

In an October 2024 fireside chat at Harvard, Altman said he “hates” ads and called the idea of combining ads with AI “uniquely unsettling,” as CNN reported. He contrasted ChatGPT’s user-aligned model with Google’s ad-driven search, saying Google’s results depended on “doing badly for the user.”

By November 2025, Altman’s position had softened. He told an interviewer he wasn’t “totally against” ads but said they would “take a lot of care to get right.” He drew a line between pay-to-rank advertising, which he said would be “catastrophic,” and transaction fees or contextual placement that doesn’t alter recommendations.

The test rolling out today follows the contextual model Altman described. Ads sit below responses and don’t affect what ChatGPT recommends. Whether that distinction holds as ad revenue grows will be the longer-term question.

Where Competitors Stand

The timing puts OpenAI’s decision in sharp contrast with its two closest rivals.

Anthropic ran a Super Bowl campaign last week centered on the tagline “Ads are coming to AI. But not to Claude.” The spots showed fictional chatbots interrupting personal conversations with sponsored pitches.

Altman called the campaign “clearly dishonest,” writing on X that OpenAI “would obviously never run ads in the way Anthropic depicts them.”

Google has also kept distance from chatbot ads. DeepMind CEO Demis Hassabis said at Davos in January that Google has no current plans for ads in Gemini, calling himself “a little bit surprised” that OpenAI moved so early. He drew a distinction between assistants, where trust is personal, and search, where Google already shows ads in AI Overviews.

That was the second time in two months that Google leadership publicly denied plans for Gemini advertising. In December, Google Ads VP Dan Taylor disputed an Adweek report claiming advertisers were told to expect Gemini ads in 2026.

The three companies are now on distinctly different paths. OpenAI is testing conversational ads at scale. Anthropic is marketing its refusal to run them. Google is running ads in AI Overviews but holding off on its standalone assistant.

Why This Matters

OpenAI says ChatGPT is used by hundreds of millions of people. CNBC reported that Altman told employees ChatGPT has about 800 million weekly users. That creates pressure to find revenue beyond subscriptions, and advertising is the proven model for monetizing free users across consumer tech.

For practitioners, today’s launch opens a new ad channel for AI platform monetization. The targeting mechanism uses conversation context rather than search keywords, which creates a different kind of intent signal. Someone asking ChatGPT for help planning a trip is further along in the decision process than someone typing a search query.

The restrictions are also worth watching. No ads near health, politics, or mental health topics means the inventory is narrower than traditional search. Combined with reported $60 CPMs and a $200K minimum, this starts as a premium play for a limited set of advertisers rather than a self-serve marketplace.

Looking Ahead

OpenAI described today’s rollout as a test to “learn, listen, and make sure we get the experience right.” No timeline was given for expanding beyond the U.S. or beyond free and Go tiers.

Separately, CNBC reported that Altman told employees in an internal Slack message that ChatGPT is “back to exceeding 10% monthly growth” and that an “updated Chat model” is expected this week.

How users respond to ads in their ChatGPT conversations will determine whether this test scales or gets pulled back. It will also test whether the distinction Altman drew in November between trust-destroying ads and acceptable contextual ones holds up in practice.