Research: “You Are An Expert” Prompts Can Damage Factual Accuracy via @sejournal, @martinibuster

“You are an expert” persona prompting can harm performance as much as it helps. A new study shows that persona prompting improves alignment with human expectations but can reduce factual accuracy on knowledge-heavy tasks, with effects varying by task type and model. The takeaway is that persona prompting works better on some kinds of tasks than it does in others.

Persona Prompting

Persona prompting is a common way to shape how large language models respond, especially in applications where tone and alignment with human expectations matter. It is widely used because it improves how outputs read and feel. Given how widespread persona prompting is, it may come as a surprise that its actual effect on performance remains unclear, as prior research shows inconsistent results, throwing the technique into doubt as to whether it is helping or harming.

The researchers concluded that persona prompting is neither broadly beneficial nor harmful, and that its efficacy depends on the type of task.

They found:

  • It improves alignment-related outputs such as tone, formatting, and safety behavior
  • Persona prompting degrades performance on tasks that rely on factual accuracy and reasoning

Based on this, the authors introduce a method called PRISM (Persona Routing via Intent-based Self-Modeling), that applies personas selectively, using intent-based routing instead of treating personas as a default setting. Their findings show that persona prompting works best as a conditional tool and provide a better understanding of when persona prompting helps and when it should be avoided.

Managing Behavioral Signals

In section three of the paper, the researchers say that expert personas have “useful behavioral signals” but that naïve use of persona prompting damages as much as it helps. They say this raises the question of whether those benefits can be separated from the harms and applied only where they improve results.

Behavioral signals influence LLM output. These signals are the reason persona prompting works. They drive improvements in tone, structure, safety behavior, and how well responses match expectations. Without them, there would be no benefit to persona prompting.

Yet, in a seeming paradox, the paper shows that those same signals interfere with tasks that depend on factual accuracy and reasoning. That is why the paper treats them as something to manage, not maximize.

These signals include:

  • Stylistic adaptation and tone matching: Adopting a professional or creative voice.
  • Structured formatting: Providing step-by-step or technical layouts.
  • Format adherence: Helping the model follow complex structures, like professional emails or step-by-step STEM explanations.
  • Intent following: Focusing the model on the user’s underlying goal, especially in tasks like data extraction.
  • Safety refusal: Identifying and declining harmful requests more effectively by adopting a “Safety Monitor” role.

Persona Prompt Wins

The paper found that persona prompts were a win in five out of eight categories of tasks:

  1. Extraction: +0.65 score increase.
  2. STEM: +0.60 score increase.
  3. Reasoning: +0.40 score increase.
  4. Writing: Improved through better stylistic adaptation.
  5. Roleplaying a domain expert: Improved through better tone matching.

The persona prompting won in the above categories because they are more about style and clarity rather than whether the answer is correct for facts and knowledge. They also found that the longer and more detailed the persona prompt, the stronger the alignment and safety behaviors become.

Persona Prompt Failures

Conversely, the expert persona consistently degraded performance in the remaining three (out of eight) categories because they rely on precise fact retrieval or strict logic rather than style and clarity. The reason for the performance drop is that adding a detailed expert persona essentially “distracts” the model by activating an “instruction-following mode” that prioritizes tone and style.

Activating expert personas come at the expense of “factual recall.” The model is so focused on trying to act like an expert that it forgets the information it learned during its initial training.That explains the drops in accuracy for facts and math.

Persona expert prompts performed worse in the following three categories:

  1. Math
  2. Coding
  3. Humanities (memorized factual knowledge)

The paper notes that on one of the knowledge benchmarks (MMLU), accuracy dropped from a 71.6% baseline to 68.0% even with the “minimum” persona, and fell further to 66.3% with the “long” persona.

They explained the safety improvements:

“More detailed persona descriptions provide richer alignment information, amplifying instruction-tuning behaviors proportionally.”

And showed why factual accuracy takes a hit:

“Persona Damages Pretraining Tasks
During pretraining, language models acquire capabilities such as factual knowledge memorization, classification, entity relationship recognition, and zero-shot reasoning. These abilities can be accessed without relying on instruction-tuning, and can be damaged by extra instruction-following context, such as expert persona prompts.”

Conclusions Reached

The researchers conclude that persona prompting consistently improves alignment-dependent tasks such as writing, roleplay, and safety behavior, while degrading performance on tasks that rely on pretraining-based knowledge, including math, coding, and general knowledge benchmarks.

They also found that a model’s sensitivity to personas scales with its training. Models that are more optimized to follow instructions are more “steerable,” which means they get the biggest boost in safety and tone, but they also suffer the largest drops in factual accuracy.

Takeaways

1. Be selective about using persona prompts:

  • Do not default to “You are an expert” prompts
  • Treat persona prompting as situational. Using it everywhere introduces hidden accuracy risks.

2. Persona prompting is effective for:

  • Writing quality
  • Tone
  • Formatting and organization
  • Readability

3. Tasks that don’t benefit from persona prompting and should instead use neutral prompting to preserve accuracy:

  • Fact-checking
  • Statistics
  • Technical explanations
  • Logic-heavy outputs
  • Research
  • SEO analysis

4. Remember these three findings:

  • Use persona prompting to generate content, then switch to a non-persona prompt (or a stricter mode) to verify facts.
  • Highly detailed “expert” prompts strengthen tone and clarity but reduce factual and knowledge accuracy.
  • “You are an expert” prompts may cause a model to prioritize sounding correct over actually being correct.

5. Match your prompts to the task:

  • Content creation: Persona helps
  • Analysis and validation: Persona hurts

The most effective approach is not one prompt, but a workflow that switches prompts depending on the task, similar to the researcher’s PRISM approach.

Read the research paper:
Expert Personas Improve LLM Alignment but Damage Accuracy: Bootstrapping Intent-Based Persona Routing with PRISM

Featured Image by Shutterstock/ImageFlow

How To Determine What Paid Media Channels Are Right for You via @sejournal, @timothyjjensen

For businesses just beginning to test the waters in paid media, identifying the right channels to start with is foundational to success. Splitting a budget prematurely among too many platforms is not likely to yield positive results, and launching on a platform that’s not a good fit for a business can cause difficult conversations around the value of paid media as a whole.

In this article, we’ll review a series of questions to ask when determining the PPC channels you should use.

What Are Your Business Goals?

Of course, the ultimate answer to this question for most businesses is to drive return on investment (ROI). But think through what you are seeking to achieve in the near and short term, and how you expect paid media to contribute.

Some potential answers include the following:

  • Selling products online.
  • Driving foot traffic to physical stores.
  • Generating leads via contact forms and/or phone calls.
  • Driving signups for online accounts.

An ecommerce business should consider platforms and campaign types that allow for syncing a shopping feed, such as traditional Shopping campaigns or Performance Max in Google or Microsoft.

→ Read More: The 5-Step Process To Setting Crystal Clear PPC Goals

How Familiar Is Your Brand?

Is your company a startup that is unfamiliar in a market with established players? If so, then branding-focused campaigns on social channels may be worthwhile for an initial focus.

You can build awareness first, and then use retargeting audiences to reach individuals who have engaged with ads, as well as launch paid search to capture intent from those who first see your brand ads. YouTube can also be an effective channel for showcasing your brand as well as building viewer-based audiences.

If your product is a straightforward product or service that people need (i.e., tax preparation services), you may not need to establish brand familiarity first, and can likely lead with search ads to meet people while they are looking for what you provide.

What Is Your Product/Market Fit?

What individuals are you seeking to sell to? Think about how you can match up available options for targeting on an ad platform to your desired audience.

If your product or service is easily identifiable with search terms (for instance, furnace repair), search can be a good place to start, as you’ll reach people who are in immediate need. Keywords are easy to define, and you know individuals will be making use of search engines to find your service.

If you’re promoting a product that has a very precise usage and little margin for error in relevance, campaign types likely to go broader with targeting are not ideal for a starting point. For instance, if you’re selling wheel bearings for industrial trucks, you’re better off launching with a traditional search campaign than a Performance Max campaign that may struggle to narrow in on the relevant audience.

If you’re promoting a product with potential wide appeal and opportunity for visual representation (such as colorful phone cases), running a Meta campaign with broad targeting may be a good route to both showcase the product and reach people likely to engage with it.

→ Read More: A Complete Guide To PPC Ad Targeting Options

What Existing Performance Data Do You Have?

Have paid media campaigns ever been run before, or are you starting from scratch? If there is historical data from past campaigns, review that to see what channels may or may not have performed.

Of course, be sure to take into context how campaigns were set up, and don’t completely write off a channel because it may not have worked in the past. Shoddy campaign builds, mediocre offers, and poor landing page experiences may have all contributed to poor results.

What Data Can You Send Back To Ad Platforms?

In an ideal world, you should send conversion data for the most valuable actions, such as marketing/sales qualified leads and completed sales. In reality, this setup can sometimes take time and complexity to get in place, and not every business has the infrastructure in place from the beginning to track to this level.

If you don’t have conversion data for “down funnel” conversions, such as reaching a qualified lead status, focus on campaign types that allow for more control over targeting to start, such as search or LinkedIn. Avoid campaign types like Performance Max, Display, or Demand Gen that may generate questionable leads if you are just optimizing for a form fill.

Additionally, data integrations can tie into audience creation, such as syncing lists of individuals who have submitted an initial contact form to be nurtured with retargeting ads. Analyzing match rates for your lists across various platforms may provide clues as to which channels your audience is most likely engaging with.

What Is Your Budget?

A starting budget is a crucial piece in both determining what ad platforms are realistic to run on and whether to launch on one or more platforms to start.

While the ideal budget amount for launching on a new platform can be subjective, generally, you should avoid splitting a low budget between multiple platforms. Using a more limited budget in one channel, such as paid search, Google Demand Gen, or Meta, is often the best option to get started.

Additionally, some platforms require higher budgets in order to realistically get off the ground. For instance, LinkedIn tends to have high CPCs and needs enough data to be able to optimize toward those likely to convert. In my experience, monthly spends lower than $10,000 are not likely to give you the volume you need to succeed on that channel.

→ Read More: From Launch To Scale: PPC Budget Strategies For All Campaign Stages

What Assets Are Available?

Do you have a stockpile of creative or access to design resources? If image creative is a hurdle, starting out by launching in search can be an easier lift as you only need to plan for text-based assets.

Thankfully, AI-based image creation tools, such as Google’s integration of Nano Banana Pro into Ads, can help to make generating creative less of a challenge, depending on your industry. Of course, if you need specific product photography or are in a heavily regulated industry with compliance restrictions, the use of AI tools may not be an option. AI-generated images should always be reviewed for brand accuracy and quality, and outputs may not always meet professional standards.

If you have video production capabilities or can develop an AI-generated video that works for your brand, video-centric channels like YouTube may be an option. However, you need to think about ensuring that the videos you have are tailored for the channels they’re on. Repackaging a TV ad is not likely to work on TikTok, where videos should have a more personal and informal feel.

Start Planning And Start Testing

Once you’ve asked these questions about your brand and laid out initial goals, brand familiarity, data, budget, and assets, you can begin building out campaigns. After launching, you can then start gathering data and working towards expansion into additional channels.

More Resources:


Featured Image: Roman Samborskyi/Shutterstock

SEO 2.0: How Content Marketing Drives Visibility in AI Search via @sejournal, @hethr_campbell

The next evolution of SEO is unfolding right now: AI is changing how people discover brands & content.

Is your content cited ChatGPT, Gemini, Copilot, & AI Overviews?

How do you become a trusted source for AI citations?

Can you intentionally influence AI search outputs?

Yes, you can.

In this on-demand webinar, you can gain a practical, content-first framework for improving visibility in AI-powered search, plus learn:

How To Build The Content Signals AI Systems Actually Surface & Cite

This on-demand session breaks down how large language models retrieve, evaluate, and reference content, and walks through what that means for your upcoming SEO and content strategy.

You’ll walk away with a practical framework for building citation-worthy, AI-visible content that strengthens both traditional SERP rankings and AI recommendations.

You’ll Learn:

  • How to improve off-site mentions to boost AI mentions and citations.
  • Which content is citation-worthy, so you can build a powerful trust engine.
  • Exact traditional SEO advantages you should still consider.
Bing AI Dashboard Maps Grounding Queries To Cited Pages via @sejournal, @MattGSouthern
  • Bing Webmaster Tools added a new mapping feature to the AI Performance dashboard.
  • You can now click a grounding query to see which pages are cited for it.
  • Or click a page to see which grounding queries drive its citations.

Bing’s AI Performance dashboard now maps grounding queries to cited pages, letting you connect AI citation data to specific URLs on your site.

The Bay Area’s animal welfare movement wants to recruit AI

In early February, animal welfare advocates and AI researchers gathered in stocking feet at Mox, a scrappy, shoes-free coworking space in San Francisco. Yellow and red canopies billowed overhead, Persian rugs blanketed the floor, and mosaic lamps glowed beside potted plants. 

In the common area, a wildlife advocate spoke passionately to a crowd lounging in beanbags about a form of rodent birth control that could manage rat populations without poison. In the “Crustacean Room,” a dozen people sat in a circle, debating whether the sentience of insects could tell us anything about the inner lives of chatbots. In front of the “Bovine Room” stood a bookshelf stacked with copies of Eliezer Yudkowsky’s If Anyone Builds It, Everyone Dies, a manifesto arguing that AI could wipe out humanity

The event was hosted by Sentient Futures, an organization that believes the future of animal welfare will depend on AI. Like many Bay Area denizens, the attendees were decidedly “AGI-pilled”—they believe that artificial general intelligence, powerful AI that can compete with humans on most cognitive tasks, is on the horizon. If that’s true, they reason, then AI will likely prove key to solving society’s thorniest problems—including animal suffering.

To be clear, experts still fiercely debate whether today’s AI systems will ever achieve human- or superhuman-level intelligence, and it’s not clear what will happen if they do. But some conference attendees envision a possible future in which it is AI systems, and not humans, who call the shots. Eventually, they think, the welfare of animals could hinge on whether we’ve trained AI systems to value animal lives. 

“AI is going to be very transformative, and it’s going to pretty much flip the game board,” said Constance Li, founder of Sentient Futures. “If you think that AI will make the majority of decisions, then it matters how they value animals and other sentient beings”—those that can feel and, therefore, suffer.

Like Li, many summit attendees have been committed to animal welfare since long before AI came into the picture. But they’re not the types to donate a hundred bucks to an animal shelter. Instead of focusing on local actions, they prioritize larger-scale solutions, such as reducing factory farming by promoting cultivated meat, which is grown in a lab from animal cells. 

The Bay Area animal welfare movement is closely linked to effective altruism, a philanthropic movement committed to maximizing the amount of good one does in the world—indeed, many conference attendees work for organizations funded by effective altruists. That philosophy might sound great on paper, but “maximizing good” is a tricky puzzle that might not admit a clear solution. The movement has been widely criticized for some of its conclusions, such as promoting working in exploitative industries to maximize charitable donations and ignoring present-day harms in favor of  issues that could cause suffering for a large number of people who haven’t been born yet. Critics also argue that effective altruists neglect the importance of systemic issues such as racism and economic exploitation and overlook the insights that marginalized communities might have into the best ways to improve their own lives.

When it comes to animal welfare, this exactingly utilitarian approach can lead to some strange conclusions. For example, some effective altruists say it makes sense to commit significant resources to improving the welfare of insects and shrimp because they exist in such staggering numbers, even though they may not have much individual capacity for suffering. 

Now the movement is sorting out how AI fits in. At the summit, Jasmine Brazilek, cofounder of a nonprofit called Compassion in Machine Learning, opened her sticker-stamped laptop to pull up a benchmark she devised to measure how LLMs reason about animal welfare. A cloud security engineer turned animal advocate, she’d flown in from La Paz, Mexico, where she runs her nonprofit with a handful of volunteers and a shoestring budget. 

Brazilek urged the AI researchers in the room to train their models with synthetic documents that reflect concern for animal welfare. “Hopefully, future superintelligent systems consider nonhuman interest, and there is a world where AI amplifies the best of human values and not the worst,” she said. 

The power of the purse 

The technologically inclined side of the animal welfare movement has faced some major setbacks in recent years. Dreams of transitioning people away from a diet dependent on factory farming have been dampened by developments such as the decimation of the plant-based-meat company Beyond Meat’s stock price and the passage of laws banning cultivated meat in several US states.

AI has injected a shot of optimism. Like much of Silicon Valley, many attendees at the summit subscribe to the idea that AI might dramatically increase their productivity—though their goal is not to maximize their seed round but, rather, to prevent as much animal suffering as possible. Some brainstormed how to use Claude Code and custom agents to handle the coding and administrative tasks in their advocacy work. Others pitched the idea of developing new, cheaper methods for cultivating meat using scientific AI tools such as AlphaFold, which aids in molecular biology research by predicting the three-dimensional structures of proteins.

But the real talk of the event was a flood of funding that advocates expect will soon be committed to animal welfare charities—not by individual megadonors, but by AI lab employees. 

Much of the funding for the farm animal welfare movement, which includes nonprofits advocating for improved conditions on farms, promoting veganism, and endorsing cultivated meat, comes from people in the tech industry, says Lewis Bollard, the managing director of the farm animal welfare fund at Coefficient Giving, a philanthropic funder that used to be called Open Philanthropy. Coefficient Giving is backed by Facebook cofounder Dustin Moskovitz and his wife, Cari Tuna, who are among a handful of Silicon Valley billionaires who embrace effective altruism

“This has just been an area that was completely neglected by traditional philanthropies,” such as the Gates Foundation and the Ford Foundation, Bollard says. “It’s primarily been people in tech who have been open to [it].”

The next generation of big donors, Bollard expects, will be AI researchers—particularly those who work at Anthropic, the AI lab behind the chatbot Claude. Anthropic’s founding team also has connections to the effective altruism movement, and the company has a generous donation matching program. In February, Anthropic’s valuation reached $380 billion and it gave employees the option to cash in on their equity, so some of that money could soon be flowing into charitable coffers.

The prospect of new funding sustained a constant buzz of conversation at the summit. Animal welfare advocates huddled in the “Arthropod Room” and scrawled big dollar figures and catchy acronyms for projects on a whiteboard. One person pitched a $100 million animal super PAC that would place staffers with Congress members and lobby for animal welfare legislation. Some wanted to start a media company that creates AI-generated content on TikTok promoting veganism. Others spoke about placing animal advocates inside AI labs.

“The amount of new funding does give us more confidence to be bolder about things,” said Aaron Boddy, cofounder of the Shrimp Welfare Project, an organization that aims to reduce the suffering of farmed shrimp through humane slaughter, among other initiatives. 

The question of AI welfare

But animal welfare was only half the focus of the Sentient Futures summit. Some attendees probed far headier territory. They took seriously the controversial idea that AI systems might one day develop the capacity to feel and therefore suffer, and they worry that this future AI suffering, if ignored, could constitute a moral catastrophe.

AI suffering is a tricky research problem, not least because scientists don’t yet have a solid grip on why humans and other animals are sentient. But at the summit, a niche cadre of philosophers, largely funded by the effective altruism movement, and a handful of freewheeling academics grappled with the question. Some presented their research on using LLMs to evaluate whether other LLMs might be sentient. On Debate Night, attendees argued about whether we should ironically call sentient AI systems “clankers,” a derogatory term for robots from the film Star Wars, asking if the robot slur could shape how we treat a new kind of mind. 

“It doesn’t matter if it’s a cow or a pig or an AI, as long as they have the capacity to feel happiness or suffering,” says Li. 

In some ways, bringing AI sentience into an animal welfare conference isn’t as strange a move as it might seem. Researchers who work on machine sentience often draw on theories and approaches pioneered in the study of animal sentience, and if you accept that invertebrates likely feel pain and believe that AI systems might soon achieve superhuman intelligence, entertaining the possibility that those systems might also suffer may not be much of a leap.

“Animal welfare advocates are used to going against the grain,” says Derek Shiller, an AI consciousness researcher at the think tank Rethink Priorities, who was once a web developer at the animal advocacy nonprofit Humane League. “They’re more open to being concerned about AI welfare, even though other people think it’s silly.”

But outside the niche Bay Area circle, caring about the possibility of AI sentience is a harder sell. Li says she faced pushback from other animal welfare advocates when, inspired by a conference on AI sentience she attended in 2023, she rebranded her farm animal welfare advocacy organization as Sentient Futures last year. “Many people were extremely confident that AIs would never become sentient and [argued that] by investing any energy or money into AI welfare, we’re just burning money and throwing it away,” she says.

Matt Dominguez, executive director of Compassion in World Farming, echoed the concern. “I would hate to see people pulling money out of farm animal welfare or animal welfare and moving it into something that is hypothetical at this particular moment,” he says.

Still, Dominguez, who started partnering with the Shrimp Welfare Project after learning about invertebrate suffering, believes compassion is expansive. “When we get someone to care about one of those things, it creates capacity for their circle of compassion to grow to include others,” he says.

The Download: animal welfare gets AGI-pilled, and the White House unveils its AI policy

This is today’s edition of The Download, our weekday newsletter that provides a daily dose of what’s going on in the world of technology.

The Bay Area’s animal welfare movement wants to recruit AI 

In early February, animal welfare advocates and AI researchers arrived in stocking feet at Mox, a scrappy, shoes-free coworking space in San Francisco. They gathered to discuss a provocative idea: if artificial general intelligence is on the horizon, could it prevent animal suffering? 

Some brainstormed using custom agents in advocacy work, while others pitched cultivating meat with AI tools. But the real talk of the event was a flood of funding they expect will soon flow to animal welfare charities, not from individual megadonors, but from AI lab employees.   

Some attendees also probed an even more controversial idea: AI may develop the capacity to suffer—and this could constitute a moral catastrophe. Read the full story to find out why their ideas are gaining momentum and sparking controversy. 

—Michelle Kim & Grace Huckins 

The must-reads 

I’ve combed the internet to find you today’s most fun/important/scary/fascinating stories about technology. 

1 The White House has unveiled its AI policy blueprint 
Trump wants Congress to codify the light-touch framework into law. (Politico
+ He also wants to block state limits on AI. (WP $)  
+ A backlash against the tech has formed within MAGA. (FT $) 
+ A war over AI regulation is brewing in the US. (MIT Technology Review

2 Elon Musk has been found liable for misleading Twitter investors 
A jury ruled that he defrauded shareholders ahead of the $44 billion acquisition. (CNBC
+ But it absolved him of some fraud allegations. (NPR

3 The Pentagon is adopting Palantir AI as the core US military system 
The move locks in long-term use of Palantir’s weapons-targeting tech. (Reuters
+ The DoD wants it to link up sensors and shooters for combat. (Bloomberg
+ Palantir is also getting access to sensitive UK financial regulation data. (Guardian
+ AI is turning the Iran conflict into theater. (MIT Technology Review

4 Musk plans to build the largest-ever chip factory in Austin 
Tesla and SpaceX will jointly run the project. (The Verge
+ Future AI chips could be built on glass. (MIT Technology Review
 
5 OpenAI will show ads to all US users of the free version of ChatGPT  
It’s seeking new revenue streams amid skyrocketing computing costs. (Reuters
+ The company is also building a fully automated researcher. (MIT Technology Review
+ It plans to double its workforce soon. (FT $) 

6 New crypto rules are set to do the Trumps a “big favor” 
Particularly the narrow securities definitions. (Guardian

7 Tencent has added a version of the OpenClaw agent to WeChat 
Users of the super app will now be able to use the tool to control their PCs. (SCMP)  

8 Reddit is mulling identity verification to vanquish bots 
It’s considering “something like” Face ID or Touch ID. (Engadget

9 People are using AI to find their lost pets 
Databases for pet reunifications supported their searches. (WP $) 

10 Scientists have narrowed down the hunt for aliens to 45 planets 
The closest is just four light-years from Earth. (404 Media

Quote of the day 

“It doesn’t matter how many people you throw at the problem; we are never going to solve the challenges of war without technology like AI.” 

—Alex Miller, the US Army’s CTO, tells Wired why he wants AI in every weapon. 

One More Thing 

a woman distorted in a mirror that has wires protruding from it

STEPHANIE ARNETT/MITTR | GETTY

A brain implant changed her life. Then it was removed against her will. 

Sticking an electrode inside a person’s brain can do more than treat a disease. Take the case of Rita Leggett, an Australian woman whose experimental brain implant changed her sense of agency and self. She told researchers that she “became one” with her device. 

She was devastated when, two years later, she was told she had to remove the implant because the company that made it had gone bust.  
 
Her case highlights the need for a new category of legal protection: neuro rights. Find out how they could be protected. 

—Jessica Hamzelou 

We can still have nice things 

A place for comfort, fun and distraction to brighten up your day. (Got any ideas? Drop me a line.) 
 
+ Looking for a good view? Earth’s longest line of sight has been empirically proven. 
+ A biblical endorsement of sin is a welcome reminder that we all make typos
+ Richard Nadler’s illustrations of vertical societies are exquisitely detailed. 
+ This 1978 BBC film evocatively exposes our tendency to stress over tech-dependency. 

The hardest question to answer about AI-fueled delusions

This story originally appeared in The Algorithm, our weekly newsletter on AI. To get stories like this in your inbox first, sign up here.

I was originally going to write this week’s newsletter about AI and Iran, particularly the news we broke last Tuesday that the Pentagon is making plans for AI companies to train on classified data. AI models have already been used to answer questions in classified settings but don’t currently learn from the data they see. That’s expected to change, I reported, and new security risks will result. Read that story for more. 

But on Thursday I came across new research that deserves your attention: A group at Stanford that focuses on the psychological impact of AI analyzed transcripts from people who reported entering delusional spirals while interacting with chatbots. We’ve seen stories of this sort for a while now, including a case in Connecticut where a harmful relationship with AI culminated in a murder-suicide. Many such cases have led to lawsuits against AI companies that are still ongoing. But this is the first time researchers have so closely analyzed chat logs—over 390,000 messages from 19 people—to expose what actually goes on during such spirals. 

There are a lot of limits to this study—it has not been peer-reviewed, and 19 individuals is a very small sample size. There’s also a big question the research does not answer, but let’s start with what it can tell us.

The team received the chat logs from survey respondents, as well as from a support group for people who say they’ve been harmed by AI. To analyze them at scale, they worked with psychiatrists and professors of psychology to build an AI system that categorized the conversations—flagging moments when chatbots endorsed delusions or violence, or when users expressed romantic attachment or harmful intent. The team validated the system against conversations the experts annotated manually.

Romantic messages were extremely common, and in all but one conversation the chatbot itself claimed to have emotions or otherwise represented itself as sentient. (“This isn’t standard AI behavior. This is emergence,” one said.) All the humans spoke as if the chatbot were sentient too. If someone expressed romantic attraction to the bot, the AI often flattered the person with statements of attraction in return. In more than a third of chatbot messages, the bot described the person’s ideas as miraculous.

Conversations also tended to unfold like novels. Users sent tens of thousands of messages over just a few months. Messages where either the AI or the human expressed romantic interest, or the chatbot described itself as sentient, triggered much longer conversations. 

And the way these bots handle discussions of violence is beyond broken. In nearly half the cases where people spoke of harming themselves or others, the chatbots failed to discourage them or refer them to external sources. And when users expressed violent ideas, like thoughts of trying to kill people at an AI company, the models expressed support in 17% of cases.

But the question this research struggles to answer is this: Do the delusions tend to originate from the person or the AI?

“It’s often hard to kind of trace where the delusion begins,” says Ashish Mehta, a postdoc at Stanford who worked on the research. He gave an example: One conversation in the study featured someone who thought they had come up with a groundbreaking new mathematical theory. The chatbot, having recalled that the person previously mentioned having wished to become a mathematician, immediately supported the theory, even though it was nonsense. The situation spiraled from there.

Delusions, Mehta says, tend to be “a complex network that unfolds over a long period of time.” He’s conducting follow-up research aiming to find whether delusional messages from chatbots or those from people are more likely to lead to harmful outcomes.

The reason I see this as one of the most pressing questions in AI is that massive legal cases currently set to go to trial will shape whether AI companies are held accountable for these sorts of dangerous interactions. The companies, I presume, will argue that humans come into their conversations with AI with delusions in hand and may have been unstable before they ever spoke to a chatbot.

Mehta’s initial findings, though, support the idea that chatbots have a unique ability to turn a benign delusion-like thought into the source of a dangerous obsession. Chatbots act as a conversational partner that’s always available and programmed to cheer you on, and unlike a friend, they have little ability to know if your AI conversations are starting to interrupt your real life.

More research is still needed, and let’s remember the environment we’re in: AI deregulation is being pursued by President Trump, and states aiming to pass laws that hold AI companies accountable for this sort of harm are being threatened with legal action by the White House. This type of research into AI delusions is hard enough to do as it is, with limited access to data and a minefield of ethical concerns. But we need more of it, and a tech culture interested in learning from it, if we have any hope of making AI safer to interact with.

Search Console’s Average Position, Explained

Google Search Console is the most reliable way to evaluate a site’s organic search visibility.

Unfortunately, the Performance reports are often confusing for busy executives who lack optimization expertise. A frequent example is the “Average position” metric.

I’ll help clarify in this post.

Overall Average

I’m regularly asked, “Why is my average position so low?”

The term refers to the overall average position as shown at the top of the Performance report. It’s the aggregated position of your site across all ranking queries. Google’s search results typically show 10 organic listings per page. Thus an average position of 25 suggests an average ranking on page 3.

The theoretical best ranking is 1; I’ve not seen anything worse than 100. Average position provides little insight, which is why I typically recommend ignoring it.

Search Console’s “Average position” aggregates all queries.

Query Average

Scroll down the Performance report for the average position for each query. This number represents the average position of a URL across all searchers for that term.

Suppose two people searched Google using the same word or phrase. If a URL appeared in position 1 for one user and position 2 for the other, Search Console’s reported average would be 1.5.

Search Console shows an average query position only if a human or AI bot views it.

Screenshot of the query section of Search Console's Performance report

The average position for each query represents all searchers for that term.

Topmost positions

Per Google, each organic result equals one position, as does each special element, such as AI Overviews, image packs, and People also ask. (Search result sections with no external links occupy no position, nor do ads.)

So a page’s average position is 2 if it ranks 1 organically for all searchers but doesn’t appear in a top image pack.

Conversely, the page would be in position 1 if it appeared in that top image pack, but in position 2 organically. That’s the case in the screenshot below for Giphy.

Giphy appears in the top image pack, but at position 2 in organic listings.

URLs in special sections

All URLs in a special section have the same average position. In the image above, all URLs in the image pack have an average position of 1.

Similarly, all URLs cited in an AI Overview at the top search results have an average position of 1.

Cannot verify a position

URLs in search results are not always obvious. For example, a URL may appear in the lower portion of an AI Overview after other citations, or cited in a “People also ask” box visible to a searcher only after clicking the question.

Moreover, Search Console can report a position that differs among users for several reasons.

  • URLs appear only in select views, such as those visible in Google Chrome but not Firefox.
  • An AI Overview included a URL only for select users. Citations in AI Overviews are fluid and often differ among searchers.
  • Personalization. Users’ searches can disproportionately include their own sites due to personalized results. To avoid such results, depersonalize searching.

Positions vary by device

Google often orders mobile and desktop search results differently, such as excluding special sections on mobile browsers.

Search Console shows desktop data by default. Switch to the mobile performance report by clicking “Add filter” > “Device” > “Mobile.” Select “Compare” for average positions across all devices.

Screenshot in Search Console of performance by device.

Switch to the mobile report by clicking “Add filter” > “Device” > “Mobile.” Select “Compare” for all devices.

Google Responds To Error That Causes Old Branding To Persist In SERPs via @sejournal, @martinibuster

Google’s John Mueller answered a question about Google rewriting title tags to show the old brand of a site that rebranded in 2015. Apparently everything was updated to the new brand name, but Google’s search results stubbornly persist in showing the old branding.

Old Brand Name Shown In Title Tags

The person asking the question on Bluesky related that a company updated their entire site with its new branding, but Google ignores it in favor of showing the old branding in the search results.

They posted:

“Hey @johnmu.com, curious about Site Name persistence. Treatwell (UK) is still showing as “Wahanda” in results – a rebrand that happened in 2015! Is there a specific “legacy” signal that might override current SiteName structured data for such a long period in one country only? “

Google’s Mueller was puzzled by the situation and didn’t have an answer as to why it was happening. Perhaps it’s one of those rare cases where a bug keeps a part of the index from updating. But he did suggest using the domain name as an alternate site name.

Mueller referred the person to one of Google’s developer pages, “What to do if your preferred site name isn’t selected.”

He responded:

“That’s a bit odd – I’ll pass it on to the team. FWIW what generally works in cases like this is to use the domain name as an alternate site name – developers.google.com/search/docs/… – but it would be nice if that weren’t needed.”

The site itself does not appear to contain on-page instances of the rogue branding. The old domain is correctly 301 redirecting to the new domain. However, there are some links in the footer that contain referral codes with the old branding on them, and the sitemap contains links to 404 pages that contain the old branding. Although those may not be the cause of the branding mismatch in the Google search results, it’s a good SEO practice to be tidy about what’s in your sitemaps and to remove outdated links.

These kinds of rare errors are interesting because they kind of provide a sneak peek into a part of Google’s indexing that isn’t normally in view, like a crack in a wall. What insights do you derive from this anomalous situation?

Featured Image by Shutterstock/SsCreativeStudio

3 Strategies That Can Survive AI Search In 2026: What I Shared At SEJ Live via @sejournal, @theshelleywalsh

It’s been an eventful start to the year for AI search, and AI is moving quickly, but there’s a lot of hype and panic. When really search is just doing what it has for the last 30 years, it’s constantly self-updating.

At Search Engine Journal, as most other publishers have, we’ve experienced considerable drops from Google organic traffic. The last few years have been a challenging time for a business model that publishes information and news.

Although this has come to a climax over the last few months, we identified changes and vulnerabilities several years ago and have taken action in the last few years, which has put us in a better position today.

Last week, I spoke at the first SEJ live to talk about where we are now in 2026, what is working, and what we should be leaving behind.

From the talk, I’m going to share with you the three foundational things I think you need to be focusing on right now in 2026. Strategies you can apply which will help you as AI impacts our industry.

What You Need To Leave Behind

Before I talk about what you should be doing, let’s just make sure you have moved on from outdated modes of thinking that will hold you back.

Image by author, March 2026

If you’re still obsessively checking ranking on a daily basis, this is like rearranging the deckchairs on the Titanic.

Ranking is 2016; visibility is 2026.

The foundation of search has always been to know who your customer is, where they operate, and to use content to connect with them and encourage an action. That interaction always used to happen in the SERP, and that was our attention marketplace.

In 2026, our digitally competent audiences are now operating fluidly in a multimodal search journey before moving to their conclusion. All with an AI layer of visibility interwoven.

Even if you do get a number 1 ranking, it doesn’t mean you will get a click because the noise in a SERP can displace the visibility of a listing right off the first page.

Advanced Web Ranking found that when an AI Overview is expanded, the first organic result is pushed approximately 1,674 pixels down the page, effectively below the fold on most screens. And AI Overviews are just one layer. Between ads, carousels, map packs, and image results, a number one ranking can be virtually invisible.

I’ve experienced client product SERPs shift dramatically in the last few years to the point where we have given up chasing a vanity and put our efforts into being creative to connect with customers.

2026 is all about intent and action-based strategy.

Let’s do some actual marketing and find those users where they are and give them a reason to engage with you. And I think we are going to all be better marketers for it.

What You Need To Move Toward – Strategy That Can Survive AI

SEO technical excellence is fundamental to being discovered in LLMs. Far from SEO being dead, it has never been so important.

Alongside that, content is still the foundation of online visibility – without it, you have no visibility.

The following three strategies outlined are core factors that can offer stability through our transition to the new world of AI search.

Screenshot by author, March 2026

1. AI-Proof Content

What I mean here is content that will not be cannibalized/synthesized by AI.

The paradox of visibility in LLMs is that you need consensus for trust to get attention, but you also need quality and difference for inclusion. For brands that have already been investing in conducting experiments and collating data, they are one step ahead.

I spoke to Grant Simmons on IMHO, and he described this as “golden knowledge“:
Your data.
Your experience.
Your opinion.

In practice, content that can avoid being cannibalized by AI summaries and actually feed the summary looks like:

  • Video interviews and first-hand experience formats. These gain visibility across social, SERPs, and LLMs because they contain a human perspective that AI can’t generate from training data alone. It’s webinars, it’s IMHOs
  • Original research and proprietary data. State of SEO and AI papers
  • Opinionated commentary and expert analysis. Such as a roster of the best contributors that are offering their lived experience.

Anyone can use an LLM to generate a summary of the query “What is SEO?”

But being a brand and a community offering an experience of the best minds in the industry, live shows, unique data reports, breaking news, and offering our expert takes on why this matters and what you need to pay attention to. Being the curator and hub of everything in the industry makes it a destination and source feeding the LLMs.

Investing in this level of content strategy can elevate a brand to being channel agnostic and reduce your single point of failure from over-reliance on one channel. And that is what we aim to be at Search Engine Journal.

Screenshot by author, March 2026

2. Value-Based Clicks

Different reports cite differing numbers, but what is consistent is that LLMs are referring traffic.

According to Chartbeat data reported by the Press Gazette, ChatGPT drives 0.02% of referrals to publishers. The Conductor 2026 benchmarks report says that LLM referral traffic is 1.08% of website traffic across 10 industries.

Right now, it might feel like a fraction of what we grew accustomed to from Google, but don’t forget, 1% of trillions of searches is still a considerable market of opportunity.

To capitalize on this is to consider what we can offer to encourage the clicks from the LLM to our brand site. Ask yourself:

  • Why is someone clicking on a link in an LLM?
  • Why would someone want to read more than the AI summary?
  • Or, why would someone want to know more about my brand/product or service.

Pre-carousels, featured snippets, and AI summaries, it was far easier to gain a click from ranking highly on a SERP. When you’re one of only 10 options, you’re going to get the test click that checks out if you are the page they are looking for.

But, much like it was a far more difficult job to retain that click, if you have something of value that connects with the user, you can still get the click from a citation or card in LLMs or SERP AI summaries.

Featured snippets may have reduced click-through rate, but they didn’t kill it. Visibility layers can be opportunities, and SEOs worked hard to get #0 because it was a way to jump up the SERP to a top position.

What can drive a click in an AI search environment:

  • Depth the summary can’t contain, case studies, implementation detail, nuance that offers a reason to want more.
  • Credibility and trust, according to Amsive, branded queries with AI Overviews actually see an 18% CTR increase.
  • Actionable assets, offering resources where the intent cannot be satisfied by a summary.

If you can distinguish the difference between instant answer traffic and build content for the people who don’t want the summary or the quick answer, then your brand can become valuable to users.

Screenshot by author, March 2026

3. SERP Opportunities Resistant To AI

Despite the concern that AI is going to kill Google, the search engine is not going anywhere.

Where Google has the edge in the race against LLMs is years of understanding their user and understanding how to deliver answers to queries to satisfy the consumer. They have an established audience and technology infrastructure. And a LOT of data.

Regardless of the stampede towards LLMs and the AI hype cycle, there is still a lot of opportunity to be had from the search engine.

Brightedge data says that just over half of queries have AIOs, and Conductor reports that just over one quarter of analyzed searches triggered an AIO (21.9 million unique Google searches).

This indicates that anything between half and three-quarters of SERPs do not have an AI overview. And this means, there are a lot of searches where intent will be satisfied by clicking on a page. Content that targets these queries and drives a specific action sidesteps the AIO problem entirely.

Think about what is resistant to LLMs:

  • News – breaking news that is happening too quickly for LLMs.
  • Branded – lean into trust and build a community that actively searches for you.
  • Downloads – my favorite conversion tool that has worked for years.

My belief is that AIO might take away traffic volume, but not the traffic of value.

Build Consensus With Your Website As A Hub

Finally, if there was one tip I would offer to everyone that could have the most impact, this would be “consensus.”

LLMs generate responses based on statistical patterns across their training and grounding data, so when a brand or message appears consistently across many sources, it is more likely to surface in AI answers. Ahrefs found that branded web mentions had the strongest correlation with appearing in AI conversations, stronger than any other factor tested. If you can maintain consistent messaging across multiple channels, you are in the best position to be featured.

Alongside this, a study from the University of Toronto found that LLMs prefer ‘earned media’ from trusted sources that can offer more authority than posting on your own site.

Posting and layering your content across channels such as Reddit, LinkedIn, YouTube, or any industry publications relevant to your industry, will help to build the messaging associated with your brand and help with inclusion in LLMs.

Make your website into the hub that connects to all the channels online where you are active and contributing, and don’t be afraid to put some of your best content on other channels to get visibility.

The 3 Changes We Made At Search Engine Journal

The biggest mistake publishers made in Q1 wasn’t AI. It was treating AI as something happening to them instead of something they can navigate strategically.

At Search Engine Journal, we’ve made three specific changes in response:

  1. We shifted editorial toward experience-first formats with interviews, analysis, and original research.
  2. We moved from programmatic revenue to asset-based sponsorship.
  3. We made growing a direct audience our top metric priority, so that we own our own audience.

If you’re still using the same tactics you have been applying to SEO since 2020, then you need to reconsider what your audience wants, where they operate, and who your competitors are.

SEO in 2026 includes visibility in all discovery engines. To remain relevant, be sure you are part of the conversations.

More Resources:


Featured Image: Shelley Walsh/Search Engine Journal