Trending Content Output Compared In ChatGPT, Gemini, And Claude via @sejournal, @gregjarboe

“Should we use ChatGPT, Gemini, or Claude to create trending content?”

That’s a good question – and one that I’m frequently asked when teaching a live class about engaging audiences through content.

You might mistakenly assume that instructors are supposed to answer questions like that. But my colleagues and I do something different.

We give our students relevant assignments that provide them with the opportunity to discover the answers to good questions for themselves.

This test-and-learn approach is something that you can also apply within your own company, agency, or team. Want to know which generative AI model to use to create trending content?

Well, conduct a test and compare the output.

Write Concise Descriptions Of The Most Emotionally Engaging Video Ads Of Euros 2024

You already know how to use Google Trends to identify search terms or topics that are trending now. In fact, many journalists use this free tool to get story ideas.

Over the long Fourth of July weekend, I used Google Trends and saw that the topic “Euros 2024” ranked No. 1 in Trending Searches with more than 1 million queries.

Next, I decided to avoid using OpenAI’s ChatGPT 3.5 for my tests. Why? Because it’s the most popular generative AI model, which means that your competitors are probably using it, too.

That means the content that this tool creates is less likely “to provide substantial value when compared to other pages in search results,” as Google’s long-standing advice recommends.

ChatGPT Output

So, I explored the custom versions of ChatGPT and selected Write For Me by puzzle.today.

Why? Because it ranked No. 1 in the Writing category. The tool also claimed it would help me “write tailored, engaging content with a focus on quality, relevance and precise word count.”

I want to create better content, not just content that is different from what everyone else is creating. So, I asked Write For Me to “write concise descriptions of the most emotionally engaging video ads of Euros 2024.”

Here’s what this generative AI tool produced:

1. Unity Through Football

 “A diverse group of children from different European countries come together on a pristine football pitch. As they play, the camera captures their joy and camaraderie, transcending language and cultural barriers. The ad concludes with a montage of professional players joining the children, emphasizing football's power to unite nations and foster friendship.”

2. Legacy of Legends

 “The ad begins with black-and-white footage of iconic moments from past Euros, highlighting legendary players. It transitions to vibrant, modern clips of current stars recreating these moments, symbolizing the passing of the torch. A heartfelt narration speaks to the continuity of dreams and the legacy carried by each generation.”

3. Homecoming

 “A montage of fans traveling from various parts of Europe to the host country, capturing their excitement and anticipation. Emotional reunions with fellow fans, old friends, and family members create a sense of belonging and community. The ad closes with a panoramic shot of a stadium filled with cheering supporters, underscoring the tournament's role in bringing people together.”

4. The Dream Begins

 “The ad follows a young aspiring footballer practicing tirelessly, inspired by their favorite Euros moments. Interspersed with clips of professional matches, it shows the child's journey from local games to a stadium filled with fans. The message is clear: dreams start small but can lead to greatness, encouraging viewers to pursue their passions.”

5. Heartbeat of the Game

 “This ad features a rhythmic sequence of football-related sounds: the thud of a ball, cheers from the crowd, and the whistle of the referee. These sounds build into a symphony, interspersed with clips of players’ intense focus and emotional celebrations. The heartbeat sound underscores the passion and excitement of the Euros, resonating with fans’ own emotions.”

6. From Streets to Stadiums

 “Children playing football in urban alleys, rural fields, and makeshift pitches across Europe are showcased. Their raw passion and joy are mirrored by professional players in grand stadiums, blurring the lines between dreams and reality. The ad concludes with a powerful message: football belongs to everyone, from the streets to the grandest arenas.

Now, it’s perfectly clear why video ads with these descriptions would be emotionally engaging, but it is frustratingly unclear which brands are running these ads during Euros 2024.

So, I’d have to do some double-checking to make sure that ChatGPT isn’t hallucinating.

Gemini Output

Next, I gave Google’s Gemini (formerly Bard) the same prompt. Here’s what this generative AI model produced:

1. Adidas: "Hey Jude" (Most Engaging)

 “This ad features rising England star Jude Bellingham alongside football legends like David Beckham and Frank Lampard. It taps into positive emotions by showcasing the passing of the torch and the enduring spirit of the game.”

2. Irn-Bru: "Get to Germany"

 “This ad, likely using humor, aims to create positive feelings and excitement around the Euros being held in Germany.”

3. Budweiser: "Bring it Home"

“This ad uses a patriotic theme, likely aiming to inspire viewers and evoke a sense of national pride for their team's chance to win the Euros.”

It’s perfectly clear which brands are running the most emotionally engaging video ads of Euros 2024, but it’s still unclear which methodology Gemini is using to make this selection.

So, I’d still have to do more due diligence to discover that it’s probably recent articles and posts about new research from DAIVID.

Claude Output

I asked Anthropic’s Claude 3.5 Sonnet to “write concise descriptions of the most emotionally engaging video ads of Euros 2024.” Here’s what this generative AI tool replied:

“I apologize, but I don't have access to specific information about video ads from Euro 2024, as my knowledge cutoff is in April 2024, and I can't browse the internet for current information.”

 “I can, however, offer some general thoughts on what makes emotionally engaging soccer/football ads:”

“Underdog stories of players or teams overcoming adversity.”
“Nostalgic montages showcasing historic Euro moments.”
“Depictions of passionate fans and their connection to the sport.”
“Inspirational messages about unity and international cooperation.”
“Cinematic recreations of iconic goals or plays.”

 “If you'd like me to expand on any of these concepts or discuss emotionally engaging sports ads in general, I'd be happy to do so. For accurate information about Euro 2024 ads, I'd recommend checking official UEFA sources or recent sports media coverage.”

Well, Claude’s honesty is admirable. But its knowledge cutoff means that I shouldn’t use this generative AI model for content about fairly recent trends.

Write Short Captions For The Funniest Photos From The 2024 Nathan’s Hot Dog Eating Contest

For my second test, I used the Google autocomplete feature to confirm that the phrase, [Nathan’s hot dog eating contest], was a trending search term.

Then, I asked each of the three generative AI tools to “write short captions for the funniest photos from the 2024 Nathan’s Hot Dog Eating Contest.”

ChatGPT Output

Here’s what Write For Me generated:

“Captions for Funniest Photos from the 2024 Nathan’s Hot Dog Eating Contest:”

"When you realize you're only halfway through your 50 hot dogs."
"Eyes on the prize: the ketchup bottle!"
"Training for this by swallowing my pride daily."
"When the bun hits you harder than the hot dog."
"Mustard not? I mustered up the courage!"
"New strategy: inhale, don’t chew."
"Hot dogs: 10, Dignity: 0."
"When you think it’s the last dog but it’s not."
"The face you make when you beat your personal best... by 1 dog."
"Calories don’t count if you don’t chew, right?"

Okay, I admit that many of these captions are funny. But it’s difficult to tell if they belong to actual photos or if this custom version of ChatGPT hallucinated the entire list.

And don’t get me started on getting permission to use any photos that exist. Of course, I could always use DALL-E 3 to turn the text above into images.

Gemini Output

Next, I gave Gemini the same assignment. Here’s what it generated:

1. Mustard Mishap: "Someone's about to be redecorated in yellow!" (Photo of a competitor with mustard splattered all over their face).

2. Relish the Moment: "This is what pure competitive joy looks like." (Photo of a competitor mid-devour with a look of intense concentration and relish dripping down their chin).

3. Bun Bungee: "Is that a hot dog or a bungee jumper? Defying the laws of physics, one frankfurter at a time." (Photo of a competitor with a hot dog halfway down their esophagus).

4. Fountain of Fury: "Looks like someone forgot their manners! (But they're probably winning.)" (Photo of a competitor with a spray of water and ketchup erupting from their mouth).

5. Doggy Downtime: "Even champions gotta rest sometimes. (Just don't let Joey Chestnut see this.)" (Photo of a competitor slumped back in their chair with a belly full of hot dogs).”

Um, okay. These are also funny captions.

But it’s hard to know if they go with actual photos that were posted on July 4, 2024, without using Google Images to conduct a series of relevant searches. And that still doesn’t give me permission to use them even if they do exist.

[Note, these responses were generated a few days before Google announced their support for labelling AI-generated images.]

Claude Output

Then, I asked Claude 3.5 Sonnet to “write short captions for the funniest photos from the 2024 Nathan’s Hot Dog Eating Contest.” Here’s what it replied:

“I apologize, but I don't have access to specific photos from the 2024 Nathan's Hot Dog Eating Contest, as my knowledge cutoff is in April 2024. Additionally, I can't generate, produce, edit, manipulate or create images.”

 “However, I can suggest some hypothetical humorous captions that might be appropriate for funny moments at a hot dog eating contest:”

"When the hot dogs fight back"
"Regretting that 50th hot dog"
"Mid-contest nap: Expert level achieved"
"Ketchup everywhere... except on the hot dogs"
"When you realize it's only been 2 minutes"
"The face of a champion... or indigestion"
"Spectators' faces: Equal parts horror and awe"
"When the bun-to-dog ratio is all wrong"
"Competitive eating: Where table manners go to die"
"That moment you question all your life choices"

 
“Remember, these are fictional captions for imaginary photos. For accurate information about the 2024 contest, you'd need to check official sources or news reports from after the event.”

Once again, Claude gets high marks for being honest about its limitations.

But unless I want to select some of these hypothetical humorous captions and use them as prompts in Adobe Firefly, then I need to use one of the other options.

Write A 400-Word Blog Post About The Top Omni Channel Retailing Trends Of 2024

For my third test, I used SparkToro to research the audience of 5,000 to 6,000 people who search for the keyword [customer retention] each month.

I used SparkToro’s trending keywords feature to analyze the search terms that have risen in popularity among searchers for “customer retention” over the last quarter.

Then, I selected the trending keyword, “omni channel retailing trends,” for this test.

ChatGPT Output

When I asked Write For Me to “write a 400-word blog post about the top omni channel retailing trends of 2024,” here’s what this generative AI tool produced:

Top Omnichannel Retailing Trends of 2024

 “As we move into 2024, the landscape of retail continues to evolve rapidly, driven by technological advancements and changing consumer expectations. Omnichannel retailing, which integrates various shopping channels into a seamless customer experience, remains at the forefront of this evolution. Here are the top omnichannel retailing trends to watch out for in 2024.”
 
1. Enhanced Personalization Through AI and Data Analytics
“In 2024, personalization goes beyond just using the customer’s name in emails. Retailers are leveraging advanced AI and data analytics to offer hyper-personalized shopping experiences. By analyzing consumer behavior, preferences, and purchase history, retailers can tailor recommendations, promotions, and content to individual customers. This level of personalization enhances customer satisfaction and increases conversion rates.”
 
2. Seamless Integration of Physical and Digital Channels 
“The line between online and offline shopping continues to blur. In 2024, more retailers are perfecting the integration of physical and digital channels. Innovations like in-store augmented reality (AR) mirrors, smart fitting rooms, and mobile app functionalities that enhance in-store experiences are becoming standard. Additionally, services like click-and-collect and in-store returns for online purchases are expected to become more streamlined and widespread.”
 
3. Expansion of Mobile Commerce (M-Commerce) 
“Mobile commerce continues to grow as consumers increasingly use their smartphones for shopping. Retailers are optimizing their mobile websites and apps to provide faster, more intuitive, and secure shopping experiences. Features like one-click purchasing, mobile wallets, and personalized push notifications are becoming essential components of mobile strategies.”

 4. Social Commerce Revolution 
“Social media platforms are transforming into powerful retail channels. In 2024, social commerce, where shopping experiences are embedded within social media, is set to explode. Platforms like Instagram, Facebook, and TikTok are enhancing their shopping functionalities, allowing users to discover, browse, and purchase products without leaving the app. Influencer marketing and user-generated content continue to drive this trend, making social media a critical part of omnichannel strategies.”
 
5. Sustainable and Ethical Retailing
“Consumers are increasingly demanding sustainability and transparency from the brands they support. In response, retailers are incorporating sustainable practices and ethical sourcing into their omnichannel strategies. This includes offering eco-friendly products, reducing packaging waste, and providing transparency about product origins and manufacturing processes. Brands that can effectively communicate their sustainability efforts through all channels will gain a competitive edge.”
 
6. Voice Commerce and Conversational AI 
“Voice-activated shopping and conversational AI are becoming more prevalent. With the rise of smart speakers and virtual assistants, consumers are using voice commands to search for products, place orders, and track deliveries. Retailers are integrating voice commerce capabilities into their omnichannel strategies to enhance convenience and improve customer engagement.”
 
Conclusion 
“In 2024, the retail industry is embracing these omnichannel trends to create more cohesive and engaging shopping experiences. By focusing on personalization, integrating physical and digital channels, leveraging mobile and social commerce, prioritizing sustainability, and adopting voice commerce, retailers can stay ahead in a competitive market. These trends highlight the importance of flexibility and innovation in meeting the evolving needs of modern consumers.”

Um, okay. This is 498 words long. This content seems designed for publication at the beginning of the year.

The blog post is missing citations from reliable sources, credible quotes, and relevant statistics, as Roger Montti’s article, “Researchers Discover How To SEO For AI Search,” noted back on Dec. 18, 2023.

So, it’s not as bad as a first draft. But I would hope that writers would learn by using this custom version of ChatGPT that they need to craft something a little bit better.

Gemini Output

Next, I asked Gemini to tackle the same assignment. Here’s what this generative AI tool produced:

The Future of Shopping: Top Omnichannel Retail Trends in 2024
 
“The retail landscape is ever-evolving, and 2024 is no exception. Consumers today expect a seamless shopping experience, regardless of whether they're browsing online, visiting a physical store, or interacting with a brand on social media. This is where omnichannel retail thrives. By creating a unified experience across all touchpoints, retailers can build stronger customer loyalty and drive sales.”
 
“Let's delve into the top omnichannel retail trends shaping the industry in 2024:”

 1. The Rise of the Data-Driven Experience:
 “With the demise of third-party cookies, retailers are focusing on collecting and utilizing first-party customer data. This data allows for real-time personalization across all channels. Imagine receiving targeted email promotions based on your recent in-store purchases, or seeing personalized product recommendations on a brand's app. Advanced analytics will not only personalize marketing efforts but also optimize search functionalities and product merchandising on e-commerce platforms.”

 2. AI Takes Center Stage: 
“Artificial intelligence (AI) is rapidly transforming the retail experience. AI-powered chatbots can answer customer queries 24/7, recommend products based on browsing history, and even assist with in-store tasks. Generative AI can create personalized marketing content at scale, while AI-driven sales automation streamlines processes for both retailers and customers.”
 
3. The Blur Between Physical and Digital: 
“The lines between online and offline shopping are continuing to blur. We're seeing a rise in ‘phygital’ experiences, such as augmented reality (AR) apps that allow customers to virtually try on clothes or visualize furniture in their homes. In-store kiosks can facilitate online purchases and allow for easy returns or exchanges. Additionally, ‘showrooming,’ where customers browse in-store and then purchase online, necessitates seamless product availability and pricing across channels.”
 
4. Social Commerce Takes Off: 
“Social media platforms are no longer just for browsing; they're becoming powerful shopping destinations. ‘Live shopping’ experiences with influencers showcasing products are gaining traction, while social media platforms are integrating smoother buying functionalities within their apps. User-generated content and social proof will continue to influence purchasing decisions, making social media a crucial touchpoint in the omnichannel journey.”

 5. Personalization Beyond Products: 
“The future of omnichannel retail goes beyond just product recommendations. Retailers are focusing on creating personalized experiences that cater to individual customer preferences. This could involve offering flexible delivery options, loyalty programs with tiered benefits, or the ability to curate in-store shopping experiences through appointments or personalized product selections.”

Conclusion: 
“In today's competitive retail environment, omnichannel isn't just a trend; it's a necessity. By embracing these trends and prioritizing a data-driven, personalized approach, retailers can not only survive but thrive in the ever-evolving landscape of commerce.”

Well, the headline is better and it’s 433 words long, which is closer to what I’d asked for. But this blog post is also missing citations from reliable sources, credible quotes, and relevant statistics.

So, I hope writers would learn the same lesson by using Gemini that they learned using a custom version of ChatGPT: It’s not bad, but a human with experience and expertise can do better.

Claude Output

Finally, I asked Claude 3.5 Sonnet to “write a 400-word blog post about the top omni channel retailing trends of 2024.” Here’s what this generative AI tool produced:

Top Omnichannel Retailing Trends Shaping 2024
 
“As we navigate through 2024, the retail landscape continues to evolve rapidly. Omnichannel strategies have become more crucial than ever, with consumers expecting seamless experiences across all touchpoints. Let's explore the key trends defining omnichannel retailing this year.”
 
1. AI-Powered Personalization: “Artificial intelligence is revolutionizing how retailers understand and cater to individual customer preferences. In 2024, we're seeing more sophisticated AI algorithms that analyze customer data across channels to deliver hyper-personalized recommendations, offers, and experiences.”
 
2. Augmented Reality Integration: “AR is bridging the gap between online and offline shopping. Retailers are leveraging AR technology to allow customers to virtually try on clothes, visualize furniture in their homes, or explore products in detail before making a purchase decision.”
 
3. Social Commerce Acceleration: “Social media platforms are increasingly becoming direct sales channels. Retailers are optimizing their presence on platforms like Instagram, TikTok, and Pinterest, enabling in-app purchases and creating shoppable content that seamlessly integrates with users' social experiences.”
 
4. Sustainable and Ethical Omnichannel Practices: “Consumers are more environmentally conscious than ever. Retailers are responding by implementing sustainable practices across their omnichannel operations, from eco-friendly packaging to transparent supply chains and ethical sourcing information available across all channels.”
 
5. Voice Commerce Integration: “With the growing adoption of smart speakers and voice assistants, retailers are optimizing their digital presence for voice search and enabling voice-activated purchases, adding another dimension to the omnichannel experience.”
 
6. Phygital Experiences: “The blend of physical and digital experiences, or ‘phygital,’ is gaining traction. Retailers are creating innovative in-store digital touchpoints and bringing elements of the physical store experience to online platforms.”
 
7. Unified Customer Data Platforms: “Retailers are investing in advanced customer data platforms that consolidate information from all channels, providing a single view of the customer and enabling more cohesive marketing and service strategies.”

8. Flexible Fulfillment Options: “The line between online and offline continues to blur with the expansion of services like buy online, pick up in-store (BOPIS), curbside pickup, and same-day delivery. Retailers are optimizing their logistics to offer customers maximum flexibility.”

9. Subscription-Based Models: “More retailers are incorporating subscription services into their omnichannel strategies, providing regular product deliveries or exclusive access to content and experiences across multiple platforms.”
 
10. Advanced Analytics for Real-Time Insights: “Retailers are leveraging advanced analytics tools to gain real-time insights into customer behavior across channels, allowing for quick adjustments to inventory, pricing, and marketing strategies.”
 
“As these trends show, the future of retail lies in creating cohesive, customer-centric experiences that seamlessly blend online and offline touchpoints. Retailers who successfully adapt to these omnichannel trends will be well-positioned to thrive in the dynamic retail environment of 2024 and beyond.”

Claude’s content is 443 words long, which is longer than Gemini’s, but shorter than the custom version of ChatGPT. And Claude generated 10 trends, while ChatGPT generated six, and Gemini generated five.

But just like the other generative AI tools, Claude’s blog post is missing citations from reliable sources, credible quotes, and relevant statistics.

So, Claude is also good at creating a rough draft. But I’d hope that writers would learn by using ChatGPT, Gemini, and Claude that they shouldn’t become a tool of their tools.

AI Is Not Going To Take Your Job

A year ago, Jensen Huang, the founder, president, and CEO of Nvidia, said,

“AI is not going to take your job. The person who uses AI is going to take your job.”

Well, a year later, we’ve learned by comparing the content output generated by ChatGPT, Gemini, and Claude that it’s still smarter to use generative AI tools to brainstorm or create a good first draft.

Then, use your experience, expertise, authoritativeness, and trustworthiness (E-E-A-T) to add emotionally engaging videos, eye-catching photos, citations from reliable sources, credible quotes, and relevant statistics to your content.

Why? Because this is the best way to provide substantial value when compared to other pages in search results.

More resources:  


Featured Image: jomel alos/Shutterstock

OpenAI GPT-4o Mini Costs Less & Wallops Competition via @sejournal, @martinibuster

OpenAI rolled out GPT-4o mini, a replacement for GPT 3.5 Turbo that is more powerful than other models in its class. Because it’s hyper efficient, GPT 4o mini will make AI available to more people at a cheaper price through better end-user applications.

GPT-4o mini

GPT-4o mini is a highly efficient version of GPT-4o that is cheaper to run and is fast. Despite it’s designation as “mini” this language model is outperforms GPT-4 and GPT-3.5 turbo, as well as solidly outperforming Google’s comparable model, Gemini Flash 1.5.

Preliminary scores by the open source Large Language Model Systems Organizations shows GPT-4o Mini outperforming Anthropic’s Claude 3 Opus and Google’s Gemini Flash 1.5 and reaching benchmark scores that are comparable to GPT 4.5 Turbo and Gemini 1.5 Pro.

Screenshot Of Language Model Scores

Cost Effective Language Model

An important feature of GPT-4o mini is that it’s cheaper to use, 60% cheaper than GPT 3.5 Turbo, which means that companies that make AI products based on OpenAI language models will be able to offer high performance AI applications that cost significantly less. This makes AI available to more people around the world.

According to OpenAI:

“Today, we’re announcing GPT-4o mini, our most cost-efficient small model. We expect GPT-4o mini will significantly expand the range of applications built with AI by making intelligence much more affordable. GPT-4o mini scores 82% on MMLU and currently outperforms GPT-41 on chat preferences in LMSYS leaderboard(opens in a new window). It is priced at 15 cents per million input tokens and 60 cents per million output tokens, an order of magnitude more affordable than previous frontier models and more than 60% cheaper than GPT-3.5 Turbo.

a text and vision model in the Assistants API, Chat Completions API, and Batch API. Developers pay 15 cents per 1M input tokens and 60 cents per 1M output tokens (roughly the equivalent of 2500 pages in a standard book). We plan to roll out fine-tuning for GPT-4o mini in the coming days.”

GPT-4o mini Availability

GPT 4o mini is available today to users of ChatGPT Free, Plus and Team, with GPT-3.5 no longer a selectable option. Enterprise users will have access next week.

Read the official announcement:

GPT-4o mini: advancing cost-efficient intelligence

Featured Image by Shutterstock/Dean Drobot

Research Confirms Google AIO Keyword Trends via @sejournal, @martinibuster

New research by enterprise search marketing company BrightEdge reveals dramatic changes to sites surfaced through Google’s AI Overviews search feature and though it maintains search market share, the data shows that AI search engine Perplexity is gaining ground at a remarkable pace.

Rapid & Dramatic Changes In AIO Triggers

The words that trigger AI Overviews are changing at an incredibly rapid pace. Some keyword trends in June may already changed in July.

AI Overviews were triggered 50% more times for keywords with the word “best” in them. But Google may have reversed that behavior because those phrases, when applied to products, don’t appear to be triggering AIOs in July.

Other AIO triggers for June 2024:

  • “What Is” keywords increased by 20% more times
  • “How to” queries increased by 15%
  • Queries with the phrase “”symptoms of” increased by about 12%
  • Queries with the word “treatment” increased by 10%

A spokesperson from BrightEdge responded to my questions about ecommerce search queries:

“AI’s prevalence in ecommerce is indeed increasing, with a nearly 20% rise in ecommerce keywords showing AI overviews since the beginning of July, and a dramatic 62.6% increase compared to the last week of June. Alongside this growth, we’re seeing a significant 66.67% uptick in product searches that contain both pros and cons from the AI overview. This dual trend indicates not only more prevalent use of AI in ecommerce search results but also more comprehensive and useful information being provided to consumers through features like the pros/cons modules.”

Google Search And AI Trends

BrightEdge used its proprietary BrightEdge Generative Parser™ (BGP) tool to identify key trends in search that may influence digital marketing for the rest of 2024. BGP is a tool that collects massive amounts of search trend data and turns it into actionable insights.

Their research estimates that each percentage point of search market share represents $1.2 billion, which means that gains as small as single digits are still incredibly valuable.

Jim Yu, founder and executive chairman of BrightEdge noted:

“There is no doubt that Google’s dominance remains strong, and what it does in AI matters to every business and marketer across the planet.

At the same time, new players are laying new foundations as we enter an AI-led multi-search universe. AI is in a constant state of progress, so the most important thing marketers can do now is leverage the precision of insights to monitor, prepare for changes, and adapt accordingly.

Google continues to be the most dominant source of search traffic, driving approximately 92% organic search referrals. A remarkable data point from the research is that AI competitors in all forms have not yet made a significant impact as a source of traffic, completely deflating speculation that AI competitors will cut into Google’s search traffic.

Massive Decrease In Reddit & Quora Referrals

Back in May 2024 Google Of interest to search marketers is that Google has followed through in reducing the amount of user generated content (UGC) surfaced through its AI Overviews search feature. UGC is responsible for many of the outrageously bad responses that generated negative press. BrightEdge’s research shows that referrals to Reddit and Quora from AI Overviews declined to “near zero” in the month of June.

Citations to Quora from AI Overviews are reported to have decreased by 99.69%. Reddit fared marginally etter in June with an 85.71% decrease

BrightEdge’s report noted:

“Google is prioritizing established, expert content over user discussions and forums.”

Bing, Perplexity And Chatbot Impact

Market share for Bing continues to increase but only by fractions of a percentage point, growing from 4.2% to 4.5%. But as they say, it’s better to be moving forward than standing still.

Perplexity on the other hand is growing at a monthly rate of 31%. Percentages however can be misleading because 31% of a relatively small number is still a relatively small number. Most publishers aren’t talking about all the traffic they’re getting from Perplexity so they still have a way to go. Nevertheless, a monthly growth rate of 31% is movement in the right direction.

Traffic from Chatbots aren’t really a thing, so this comparison should be put into that perspective. Sending referral traffic to websites isn’t really what chatbots like Claude and ChatGPT are about (at this point in time). The data shows that both Claude and ChatGPT are not sending much traffic.

OpenAI however is hiding referrals from the websites that it’s sending traffic to which makes it difficult to track it. Therefore a full understanding of the impact of LLM traffic, because ChatGPT uses a rel=noreferrer HTML attribute which hides all traffic originating from ChatGPT to websites. The use of the rel=noreferrer link attribute is not unusual though because it’s an industry standard for privacy and security.

BrightEdge’s analysis looks at this from a long term perspective and anticipates that referral traffic from LLMs will become more prevalent and at some point will become a significant consideration for marketers.

This is the conclusion reached by BrightEdge:

“The overall number of referrals from LLMs is small and expected to have little industry impact at this time. However, if this incremental growth continues, BrightEdge predicts it will influence where people search online and how brands approach optimizing for different engines.”

Before the iPhone existed, many scoffed at the idea of the Internet on mobile devices. So BrightEdge’s conclusions about what to expect from LLMs are not unreasonable.

AIO trends have already changed in July, pointing to the importance of having fresh data for adapting to fast changing AIO keyword trends.  BrightEdge delivers real-time data updated on a daily basis so that marketers can make better informed decisions.

Understand AI Overview Trends:

Ten Observations On AI Overviews For June 2024

Featured Image by Shutterstock/Krakenimages.com

Anthropic Announces Free Claude AI Chatbot For Android via @sejournal, @martinibuster

Anthropic announced the release of a new Claude Android app that uses their powerful Claude 3.5 Sonnet language model. The app is available free (with usage limits) and also with paid plans.

Anthropic Claude

Claude is a powerful AI chatbot that offers advanced reasoning, can do real-time image analysis, and can translate languages in real-time. Claude 3.5 Sonnet is Anthropic’s most advanced language model, introduced in late June 2024.

According to Anthropic:

“Claude 3.5 Sonnet raises the industry bar for intelligence, outperforming competitor models and Claude 3 Opus on a wide range of evaluations, with the speed and cost of our mid-tier model, Claude 3 Sonnet.

Claude 3.5 Sonnet sets new industry benchmarks for graduate-level reasoning (GPQA), undergraduate-level knowledge (MMLU), and coding proficiency (HumanEval). It shows marked improvement in grasping nuance, humor, and complex instructions, and is exceptional at writing high-quality content with a natural, relatable tone.”

Claude By Anthropic Android App

The Claude AI chatbot app is currently available for iOS and now it’s available from the Google Play store for Android users. Downloading and signing up is easy. Once signed in and verified users can start using Claude absolutely free. I downloaded it and gave it a try and was pleasantly surprised at its ability to help create a ramen recipe from scratch. A cool feature of the app is that it can continue chats from other devices.

The official announcement described various ways it’s useful:

“Use Claude for work or for fun. Whether you’re drafting a business proposal between meetings, translating menus while traveling, brainstorming gift ideas while shopping, or composing a speech while waiting for a flight, Claude is ready to assist you.”

Download the Claude by Anthropic Android App from Google Play:

Claude by Anthropic

Read the official announcement:

Claude Android app

Unlocking The Power Of LLM And Knowledge Graph (An Introduction)

We are in an exciting era where AI advancements are transforming professional practices.

Since its release, GPT-3 has “assisted” professionals in the SEM field with their content-related tasks.

However, the launch of ChatGPT in late 2022 sparked a movement towards the creation of AI assistants.

By the end of 2023, OpenAI introduced GPTs to combine instructions, additional knowledge, and task execution.

The Promise Of GPTs

GPTs have paved the way for the dream of a personal assistant that now seems attainable. Conversational LLMs represent an ideal form of human-machine interface.

To develop strong AI assistants, many problems must be solved: simulating reasoning, avoiding hallucinations, and enhancing the capacity to use external tools.

Our Journey To Developing An SEO Assistant

For the past few months, my two long-time collaborators, Guillaume and Thomas, and I have been working on this topic.

I am presenting here the development process of our first prototypal SEO assistant.

An SEO Assistant, Why?

Our goal is to create an assistant that will be capable of:

  • Generating content according to briefs.
  • Delivering industry knowledge about SEO. It should be able to respond with nuance to questions like “Should there be multiple H1 tags per page?” or “Is TTFB a ranking factor?”
  • Interacting with SaaS tools. We all use tools with graphical user interfaces of varying complexity. Being able to use them through dialogue simplifies their usage.
  • Planning tasks (e.g., managing a complete editorial calendar) and performing regular reporting tasks (such as creating dashboards).

For the first task, LLMs are already quite advanced as long as we can constrain them to use accurate information.

The last point about planning is still largely in the realm of science fiction.

Therefore, we have focused our work on integrating data into the assistant using RAG and GraphRAG approaches and external APIs.

The RAG Approach

We will first create an assistant based on the retrieval-augmented generation (RAG) approach.

RAG is a technique that reduces a model’s hallucinations by providing it with information from external sources rather than its internal structure (its training). Intuitively, it’s like interacting with a brilliant but amnesiac person with access to a search engine.

Architecture of a Retrieval-Augmented Generation (RAG) assistantImage from author, June 2024

To build this assistant, we will use a vector database. There are many available: Redis, Elasticsearch, OpenSearch, Pinecone, Milvus, FAISS, and many others. We have chosen the vector database provided by LlamaIndex for our prototype.

We also need a language model integration (LMI) framework. This framework aims to link the LLM with the databases (and documents). Here too, there are many options: LangChain, LlamaIndex, Haystack, NeMo, Langdock, Marvin, etc. We used LangChain and LlamaIndex for our project.

Once you choose the software stack, the implementation is fairly straightforward. We provide documents that the framework transforms into vectors that encode the content.

There are many technical parameters that can improve the results. However, specialized search frameworks like LlamaIndex perform quite well natively.

For our proof-of-concept, we have given a few SEO books in French and a few webpages from famous SEO websites.

Using RAG allows for fewer hallucinations and more complete answers. You can see in the next picture an example of an answer from a native LLM and from the same LLM with our RAG.

RAG LLM versus Native LLM : which one is better?Image from author, June 2024

We see in this example that the information given by the RAG is a little bit more complete than the one given by the LLM alone.

The GraphRAG Approach

RAG models enhance LLMs by integrating external documents, but they still have trouble integrating these sources and efficiently extracting the most relevant information from a large corpus.

If an answer requires combining multiple pieces of information from several documents, the RAG approach may not be effective. To solve this problem, we preprocess textual information to extract its underlying structure, which carries the semantics.

This means creating a knowledge graph, which is a data structure that encodes the relationships between entities in a graph. This encoding is done in the form of a subject-relation-object triple.

In the example below, we have a representation of several entities and their relationships.

Example of a knowledge graphImage from author, June 2024

The entities depicted in the graph are “Bob the otter” (named entity), but also “the river,” “otter,” “fur pet,” and “fish.” The relationships are indicated on the edges of the graph.

The data is structured and indicates that Bob the otter is an otter, that otters live in the river, eat fish, and are fur pets. Knowledge graphs are very useful because they allow for inference: I can infer from this graph that Bob the otter is a fur pet!

Building a knowledge graph is a task that has been done for a long time with NLP techniques. However LLMs facilitate the creation of such graphs thanks to their capacity to process text. Therefore, we will ask an LLM to create the knowledge graph.

From text to knowledge graph triplesImage from author, June 2024

Of course, it’s the LMI framework that efficiently guides the LLM to perform this task. We have used LlamaIndex for our project.

Furthermore, the structure of our assistant becomes more complex when using the graphRAG approach (see next picture).

Architecture of a RAG + graphRAG + APIs assistantImage from author, June 2024

We will return later to the integration of tool APIs, but for the rest, we see the elements of a RAG approach, along with the knowledge graph. Note the presence of a “prompt processing” component.

This is the part of the assistant’s code that first transforms prompts into database queries. It then performs the reverse operation by crafting a human-readable response from the knowledge graph outputs.

The following picture shows the actual code we used for the prompt processing. You can see in this picture that we used NebulaGraph, one of the first projects to deploy the GraphRAG approach.

Actual code used for the prompt processingImage from author, June 2024

One can see that the prompts are quite simple. In fact, most of the work is natively done by the LLM. The better the LLM, the better the result, but even open-source LLMs give quality results.

We have fed the knowledge graph with the same information we used for the RAG. Is the quality of the answers better? Let’s see on the same example.

Example answer from a graphRAG assistantImage from author, June 2024

I let the reader judge if the information given here is better than with the previous approaches, but I feel that it is more structured and complete. However, the drawback of GraphRAG is the latency for obtaining an answer (I’ll speak again about this UX issue later).

Integrating SEO Tools Data

At this point, we have an assistant that can write and deliver knowledge more accurately. But we also want to make the assistant able to deliver data from SEO tools. To reach that goal, we will use LangChain to interact with APIs using natural language.

This is done with functions that explain to the LLM how to use a given API. For our project, we used the API of the tool babbar.tech (Full disclosure: I am the CEO of the company that develops the tool.)

A langchain functionImage from author, June 2024

The image above shows how the assistant can gather information about linking metrics for a given URL. Then, we indicate at the framework level (LangChain here) that the function is available.

tools = [StructuredTool.from_function(get_babbar_metrics)]
agent = initialize_agent(tools, ChatOpenAI(temperature=0.0, model_name="gpt-4"), 
agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION, verbose=False, memory=memory)

These three lines will set up a LangChain tool from the function above and initialize a chat for crafting the answer regarding the data. Note that the temperature is zero. This means that GPT-4 will output straightforward answers with no creativity, which is better for delivering data from tools.

Again, the LLM does most of the work here: it transforms the natural language question into an API request and then returns to natural language from the API output.

LLM together with external APIsImage from author, June 2024

You can download Jupyter Notebook file with step by step instructions and build GraphRAG conversational agent on your local enviroment.

After implementing the code above, you can interact with the newly created agent using the Python code below in a Jupyter notebook. Set your prompt in the code and run it.

import requests
import json

# Define the URL and the query
url = "http://localhost:5000/answer"

# prompt 
query = {"query": "what is seo?"}

try:
    # Make the POST request
    response = requests.post(url, json=query)
    
    # Check if the request was successful
    if response.status_code == 200:
        # Parse the JSON response
        response_data = response.json()
        
        # Format the output
        print("Response from server:")
        print(json.dumps(response_data, indent=4, sort_keys=True))
    else:
        print("Failed to get a response. Status code:", response.status_code)
        print("Response text:", response.text)
except requests.exceptions.RequestException as e:
    print("Request failed:", e)

It’s (Almost) A Wrap

Using an LLM (GPT-4, for instance) with RAG and GraphRAG approaches and adding access to external APIs, we have built a proof-of-concept that shows what can be the future of automation in SEO.

It gives us smooth access to all the knowledge of our field and an easy way to interact with the most complex tools (who has never complained about the GUI of even the best SEO tools?).

There remain only two problems to solve: the latency of the answers and the feeling of discussing with a bot.

The first issue is due to the computation time needed to go back and forth from the LLM to the graph or vector databases. It could take up to 10 seconds with our project to obtain answers to very intricate questions.

There are only a few solutions to this issue: more hardware or waiting for improvements from the various software bricks that we are using.

The second issue is trickier. While LLMs simulate the tone and writing of actual humans, the fact that the interface is proprietary says it all.

Both problems can be solved with a neat trick: using a text interface that is well-known, mostly used by humans, and where latency is usual (because used by humans in an asynchronous way).

We chose WhatsApp as a communication channel with our SEO assistant. This was the easiest part of our work, done using the WhatsApp business platform through Twilio’s Messaging APIs.

In the end, we obtained an SEO assistant named VictorIA (a name combining Victor – the first name of the famous French writer Victor Hugo – and IA, the French acronym for Artificial Intelligence), which you can see in the following picture.

Screenshots of the final assistant on whatsappImage from author, June 2024

Conclusion

Our work is just the first step in an exciting journey. Assistants could shape the future of our field. GraphRAG (+APIs) boosted LLMs to enable companies to set up their own.

Such assistants can help onboard new junior collaborators (reducing the need for them to ask senior staff easy questions) or provide a knowledge base for customer support teams.

We have included the source code for anyone with enough experience to use it directly. Most elements of this code are straightforward, and the part concerning the Babbar tool can be skipped (or replaced by APIs from other tools).

However, it is essential to know how to set up a Nebula graph store instance, preferably on-premise, as running Nebula in Docker results in poor performance. This setup is documented but can seem complex at first glance.

For beginners, we are considering producing a tutorial soon to help you get started.

More resources: 


Featured Image: sdecoret/Shutterstock

GraphRAG Is A Better RAG And Now It’s Free via @sejournal, @martinibuster

Microsoft is making publicly available a new technology called GraphRAG, which enables chatbots and answer engines to connect the dots across an entire dataset, outperforming standard Retrieval-Augmented Generation (RAG) by large margins.

What’s The Difference Between RAG And GraphRAG?

RAG (Retrieval-Augmented Generation) is a technology that enables an LLM to reach into a database like a search index and use that as a basis for answering a question. It can be used to bridge a large language model and a conventional search engine index.

The benefit of RAG is that it can use authoritative and trustworthy data in order to answer questions. RAG also enables generative AI chatbots to use up to date information to answer questions about topics that the LLM wasn’t trained on. This is an approach that’s used by AI search engines like Perplexity.

The upside of RAG is related to its use of embeddings. Embeddings is a way of representing the semantic relationships between words, sentences, and documents. This representation enables the retrieval part of RAG to match a search query to text in a database (like a search index).

But the downside of using embeddings is that it limits the RAG to matching text at a granular level (as opposed to a global reach across the data).

Microsoft explains:

“Since naive RAG only considers the top-k most similar chunks of input text, it fails. Even worse, it will match the question against chunks of text that are superficially similar to that question, resulting in misleading answers.”

The innovation of GraphRAG is that it enables an LLM to answer questions based on the overall dataset.

What GraphRAG does is it creates a knowledge graph out of the indexed documents, also known as unstructured data. The obvious example of unstructured data are web pages. So when GraphRAG creates a knowledge graph, it’s creating a “structured” representation of the relationships between various “entities” (like people, places, concepts, and things) which is then more easily understood by machines.

GraphRAG creates what Microsoft calls “communities” of general themes (high level) and more granular topics (low level). An LLM then creates a summarization of each of these communities, a “hierarchical summary of the data” that is then used to answer questions. This is the breakthrough because it enables a chatbot to answer questions based more on knowledge (the summarizations) than depending on embeddings.

This is how Microsoft explains it:

“Using an LLM to summarize each of these communities creates a hierarchical summary of the data, providing an overview of a dataset without needing to know which questions to ask in advance. Each community serves as the basis of a community summary that describes its entities and their relationships.

…Community summaries help answer such global questions because the graph index of entity and relationship descriptions has already considered all input texts in its construction. Therefore, we can use a map-reduce approach for question answering that retains all relevant content from the global data context…”

Examples Of RAG Versus GraphRAG

The original GraphRAG research paper illustrated the superiority of the GraphRAG approach in being able to answer questions for which there is no exact match data in the indexed documents. The example uses a limited dataset of Russian and Ukrainian news from the month of June 2023 (translated to English).

Simple Text Matching Question

The first question that was used an example was “What is Novorossiya?” and both RAG and GraphRAG answered the question, with GraphRAG offering a more detailed response.

The short answer by the way is that “Novorossiya” translates to New Russia and is a reference to Ukrainian lands that were conquered by Russia in the 18th century.

The second example question required that the machine make connections between concepts within the indexed documents, what Microsoft calls a “query-focused summarization (QFS) task” which is different than a simple text-based retrieval task. It requires what Microsoft calls, “connecting the dots.”

The question asked of the RAG and GraphRAG systems:

“What has Novorossiya done?”

This is the RAG answer:

“The text does not provide specific information on what Novorossiya has done.”

GraphRAG answered the question of “What has Novorossiya done?” with a two paragraph answer that details the results of the Novorossiya political movement.

Here’s a short excerpt from the two paragraph answer:

“Novorossiya, a political movement in Ukraine, has been involved in a series of destructive activities, particularly targeting various entities in Ukraine [Entities (6494, 912)]. The movement has been linked to plans to destroy properties of several Ukrainian entities, including Rosen, the Odessa Canning Factory, the Odessa Regional Radio Television Transmission Center, and the National Television Company of Ukraine [Relationships (15207, 15208, 15209, 15210)]…

…The Office of the General Prosecutor in Ukraine has reported on the creation of Novorossiya, indicating the government’s awareness and potential concern over the activities of this movement…”

The above is just some of the answer which was extracted from the limited one-month dataset, which illustrates how GraphRAG is able to connect the dots across all of the documents.

GraphRAG Now Publicly Available

Microsoft announced that GraphRAG is publicly available for use by anybody.

“Today, we’re pleased to announce that GraphRAG is now available on GitHub, offering more structured information retrieval and comprehensive response generation than naive RAG approaches. The GraphRAG code repository is complemented by a solution accelerator, providing an easy-to-use API experience hosted on Azure that can be deployed code-free in a few clicks.”

Microsoft released GraphRAG in order to make the solutions based on it more publicly accessible and to encourage feedback for improvements.

Read the announcement:

GraphRAG: New tool for complex data discovery now on GitHub

Featured Image by Shutterstock/Deemerwha studio

AIO Pullback: Google Shows 2/3 Fewer AIOs And More Citations via @sejournal, @Kevin_Indig

It has become quiet around AI Overviews. One month after my initial traffic impact analysis, I updated my data for AIOs. The results are important for anyone who aims for organic traffic from Google as we’re seeing a shift in AIO structures.

Shortly after Google just launched AI Overviews on May 14, I looked at 1,675 queries and found:

  • -8.9% fewer organic clicks when a domain is cited in AIOs than regular results.
  • A strong relationship between a domain’s organic ranks and AIO citations.
  • Variations of referral traffic depending on user intent.

Since then:

  • Featured snippets and AIOs confuse users with slightly different answers.
  • Google has significantly pulled back AIOs across all industries.
  • AIOs cite more sources.

Boost your skills with Growth Memo’s weekly expert insights. Subscribe for free!

AIOs Dropped By Two-Thirds

A few days after Google launched AIOs in the US, users found misleading and borderline harmful answers.

In a post titled “About last week,” VP of Search Liz Reid addressed the issue, but also called out that many queries were phrased in a way that would likely return questionable answers.

The debate about LLM answers and questionable queries is not new. Yes, you might get a funny answer when you ask an LLM a funny question. Leading queries were used in the NY Times vs. OpenAI lawsuit and backlash against Perplexity and are no different than leading questions that suggest the answer.

After the PR backlash, Google dropped AIOs across almost every industry by an average of two-thirds.

  • May 30: 0.6% on desktop, 0.9% on mobile.
  • June 28: 0.2% on desktop, 0.3% on mobile.

Industries with the largest drops (data from Semrush Sensor):

  • Health: -3.7% desktop, 1.3% mobile.
  • Science: -1% desktop, -2.6% mobile.
  • People & Society: -2% desktop, -3.9% mobile.
Bar chart showing percentage changes in web traffic across various categoriesImage Credit: Kevin Indig

It seems that YMYL industries, such as health, science, animals, and law, were most affected. Some industries gained a small amount of AIOs, but not more than a negligible 0.2%.

  • Example: SEOmonitor clearly shows the pullback in visibility metrics for the jobs site monster.com.
A line graph with two lines indicating visibility percentages over time, showing a red arrow highlighting a peak around mid-June. The time period from May 1 to June 22 reflects the impact of Google AIOs on these trends.Image Credit: Kevin Indig

For the 1,675 queries I analyzed, the number of AIOs dropped from 42% to 23% of queries (almost half). Interestingly, the domain was cited more often (31% vs. 25%, more shortly) and ranked more often in the top 10 spots (45% vs. 41%).

Bar chart showing changes in AIOs in health before and after Google PR backlash. Metrics compared: AIOs shown, Citations, and Domain ranks in top 10, with percentages for 6/29 and 5/23. This illustrates the AIO Pullback during this period.Image Credit: Kevin Indig

Queries that stopped showing AIOs had, on average, less search volume. However, I couldn’t detect a clear pattern across word count, user intent, or SERP features for queries that gained vs. lost AIOs. The effect applies broadly, meaning Google reduced AIOs for all types of queries.

A bar chart titled Image Credit: Kevin Indig

AIOs Lean Heavily On No. 1 Web Result For Text Snippets

The before and after comparison allows us to learn more about the structure and behavior of AIOs.

For example, [hair growth products] and [best hair growth products] deliver almost identical AIOs (see screenshots below). The text is the same, but the product list and cited sources are slightly different. Google treats product searches as equal to “best” searches (makes sense).

SERPs for hair growth products (Image Credit: Kevin Indig)
SERPs for best hair growth products (AIO text is identical to screenshot above) Image Credit: Kevin Indig

The biggest difference is that the query for [hair growth products] shows no citation carousel on the side when you click the “show more” button (another example below).

On mobile, the carousel lives at the bottom of the AIO, which is not great for click-throughs. These subtle design differences likely make a big difference when it comes to clicks from AIOs since more prominently featured citations increase the likelihood of clicks.

Citations only expand when users click “show more” (Image Credit: Kevin Indig)

For transactional queries like [hair growth products], Google ranks products in the AIO in no apparent order.

I cross-referenced reviews, average ratings, price, organic product carousel and references in top-ranking articles – none indicate a relationship with the ranking in the AIO. It seems Google leans on its Shopping Graph to sort product lists.

To structure the AIO text, Google seems to pick more elements from the organic No. 1 result than others. For example, time.com ranks No. 1 for [best hair growth products]. Even though the citation in the AIO highlights a section about ingredients (purple in the screenshot below), the whole text closely mirrors the structure of the TIME article before it lists products.

The AIO mirrors the text on the No. 1 web result (time.com) (Image Credit: Kevin Indig)

AIOs use fragments of top web results because LLMs commonly use Retrieval Augmented Generation (RAG) to generate answers.

I wrote in How SEO might thrive under Bard and Prometheus:

Sridhar says that Neeva uses a technique called Retrieval Augmented Generation (RAG), a hybrid of classic information retrieval and machine learning. With RAG, you can train LLMs (Large Language Models) through documents and “remove” inaccurate results by setting constraints. In plain terms, you can show AI what you want with the ranking score for web pages. That seems to be the same or similar technique Bing uses to make sure Prometheus results are as accurate and relevant as possible.

The best example of Google mirroring the AIO after the No. 1 web result (in some cases) is the answer for [rosemary oil for hair growth]. The AIO pulls its text from MedicalNewsToday (No. 1) and restructures the answer.

Highlighted text comparing studies from 2015 and 2022 on the effectiveness of rosemary oil for hair growth, noting its potential treatment for androgenic alopecia similar to Minoxidil after 6 weeks of use, cited fewer AIOs overall.Text in the AI Overview vs. a snippet from MedicalNewsToday (Image Credit: Kevin Indig)

AIOs And Featured Snippets Still Co-Exist

For more informational queries with a featured snippet, like [dht], [panic attack vs. anxiety attack], or [does creatine cause hair loss], Google closely mirrors the answer in the featured snippets and elaborates further.

High overlap between AIOs and featured snippets (Image Credit: Kevin Indig)

In some cases, the elaboration might confuse users. When searching for [which vitamin deficiency causes hair loss], users see a long list in the AIO and a single answer in the featured snippet. While not contradicting each other, the AIO answer makes the featured snippet seem less trustworthy.

A Google search result page detailing information on vitamins and hair loss, with citations emphasizing that Vitamin D deficiency can lead to dry, brittle hair prone to breaking easily and increased shedding and thinning.Image Credit: Kevin Indig

In my opinion, Google would be best off not showing a featured snippet when an AIO is present. However, that would be bad news for sites ranking in featured snippets.

AIOs Contain More Citations

One way Google seems to have increased the accuracy of AIOs after the PR backlash is by adding more citations. The average number of citations increased from 15 to 32 in the sample of 1,675 keywords I analyzed. I haven’t yet been able to confirm that more citations are used to compile the answer, but more outgoing links to webpages are a good signal for the open web because they increase the chance of getting click-throughs from AIOs.

Both Reddit and Wikipedia were cited more often after the PR Backlash. I counted citations from those two domains because marketers pay a lot of attention to influencing the public discourse on Reddit, while Wikipedia has a reputation for having more gatekeepers.

Bar chart comparing citations from Reddit and Wikipedia in Google AI Overviews on two dates. Image Credit: Kevin Indig

Keep in mind that, with 0.8% and 1%, the number of citations is relatively low. It seems AIO heavily diversifies the number of citations. Only 23 keywords in the 1,675 keyword sample returned more than 10% of citations from Reddit after the PR backlash (28 for Wikipedia).

Accountability

We can conclude that:

  1. Google shows 50-66% fewer AIOs, which reduces the risk of losing organic traffic – for now.
  2. There seem to be more opportunities to be cited in AIOs, but strong performance in classic web search still largely determines citations and referral clicks from AIOs.
  3. Featured snippets get fewer clicks when AIOs are present since they elaborate much more on the answer.

Google becomes more accountable as it touches the border to publishing with AI Overviews. Verticals like health, science, and law continuously morph as new evidence comes out. It will be curious to understand whether AIOs are able to factor new evidence and opinions in and at what speed.

It’s not clear how, exactly, AI Overviews evaluate the strength of evidence, or whether it takes into account contradictory research findings, like those on whether coffee is good for you. “Science isn’t a bunch of static facts,” Dr. Yasmin said. She and other experts also questioned whether the tool would draw on older scientific findings that have since been disproved or don’t capture the latest understanding of an issue.

If AIOs adapt to new information, websites need to monitor AIOs and adapt content at an equal speed. The adaptation challenge alone will provide room for competitive advantages.


AI Overviews: About last week

Google Is Using A.I. to Answer Your Health Questions. Should You Trust It?


Featured Image: Paulo Bobita/Search Engine Journal

New Features Improve Anthropic’s Claude Collaborative Workflow via @sejournal, @martinibuster

Anthropic announced new features that will significantly enhance Claude’s functionality to make it more collaborative, easier to use and speed up workflows. The new functionalities enable teams to ground Claude with the documentation needed to complete tasks, brainstorm, and be able to get things done faster with AI.

Three Major Steps Forward

The improvements that Anthropic is introducing cover three areas:

1. Projects:
A place to organize chats and knowledge.

2. Sharing:
Better collaboration with teams

3. Artifacts:
This has already rolled out, it’s a collaborative workspace for creating and editing content, coding, and designing with Claude in real-time.

1. Projects

Anthropic’s Projects is a collaborative space where team members can share curated chats and knowledge together in order to enable better decisions and brainstorming. All Claude.AI Pro and Team subscribers will have access to Projects.

Each project has a 200K context window for documents, code and other data that can be used to improve output.

According to Anthropic:

“Projects allow you to ground Claude’s outputs in your internal knowledge—be it style guides, codebases, interview transcripts, or past work. This added context enables Claude to provide expert assistance across tasks, from writing emails like your marketing team to writing SQL queries like a data analyst. With Projects, you can get started much faster and extend your skills further for any task.”

With Projects, a team can upload documents that provide the knowledge necessary for completing tasks, such as legal documentation, course material, historical financial reports and economic indicators, virtually any documentation that Claude can use for analysis or content creation.

2. Sharing

This is a way for team members to share relevant and important chats with each other through a shared activity feed. Anthropic envisions Sharing as especially useful for creative projects, research, and product development. For example, it’s a way to share brain-storming sessions and for web designers and other stakeholders to share ideas and work together with Claude to complete projects.

3. Artifacts

Artifacts is a way to create together with Claude, with a user interface that shows the chat on one side and the output on the other.

Anthropic shares five examples of how Artifacts can be used:

“1. Software Development: Programmers can use Artifacts to collaboratively write and debug code. Claude can help generate code snippets and provide real-time explanations.

2. Graphic Design: Designers can work with Claude to create and refine SVG graphics for logos or illustrations, iterating on designs in real-time.

3. Marketing: Content creators can use Artifacts to draft and edit marketing copy. Claude can suggest improvements and generate alternative versions side-by-side.

4. Data Analysis: Data scientists can collaborate with Claude to write and optimize SQL queries, visualizing data in charts and trendlines, and refining analyses together.

5. UX/UI Design: Designers can work with Claude to prototype website layouts using HTML and CSS, previewing changes instantly in the Artifacts window.”

This is Just The Beginning

Anthropic shared that they will be rolling out additional features such as integrations with popular third-party apps, further extending Claude for AI-assisted collaboration.

Read more from Anthropic’s announcement.

Featured Image by Shutterstock/Photo For Everything

User-Directed AI Content: On Perplexity, Users Can Direct AI To Create Content via @sejournal, @Kevin_Indig

Boost your skills with Growth Memo’s weekly expert insights. Subscribe for free!

Perplexity’s strategy behind its new Pages feature created a deep rift with publishers, but the reaction seems blown out of proportion. It’s much more interesting as a case study for user-directed AI content (UDC instead of UGC).

Perplexity Pages allows users to “create beautifully designed, comprehensive articles on any topic.” You can turn a thread, a prompt sequence, into a page about a topic.

As a regular Growth Memo reader, you quickly grasp that this is a growth strategy where, ideally, users create AI content that ranks in organic search and brings visitors to perplexity.ai that converts into paying subscribers.

The growth strategy fits into what CEO Srinivas explains as “an aggregator of information.” It holds power by providing a superior user experience, which allows it to channel demand and commoditize supply.

Drop In The Bucket

When we look at actual data, we can see that the media reaction is overblown. Not in the critique but in impact. It’s fair to ask Perplexity to adjust attribution, follow web standards like robots.txt, and use official IPs like search engines do as well.

According to developer Ryan Knight, Perplexity crawls the web with a headless browser that masks its IP string.

CEO Srinivas said Perplexity obeys robots.txt, and the masked IP came from a third-party service. But he also mentioned that “the emergence of AI requires a new kind of working relationship between content creators, or publishers, and sites like his.”

But in terms of benefit for Perplexity, Pages is a drop in the bucket.

Image Credit: Kevin Indig

91% of organic traffic to perplexity.ai comes from branded terms like “perplexity.”

Only 47,000 out of 217,000 (21.6%) monthly visitors to Pages come from organic, non-branded keywords globally.

In the US, it’s 55% (20,000/36,000). However, compared to x monthly visits from branded terms, Pages doesn’t make a dent in Perplexity’s organic traffic.

Image Credit: Kevin Indig

In reality, most traffic to Perplexity comes through its brand and word of mouth. The recent media coverage might have helped Perplexity more than it harmed. The site has hit new all-time traffic highs every day since January 2024, according to Similarweb.

Perplexity’s whole domain has only 950 pages, of which Pages make up almost 600. Compared to other sites – like Wikipedia’s 6.8 million articles on the English version alone – that’s just not a lot. Stronger scale effects will emerge as Pages get more traction. Right now, Pages is a nascent beta feature.

Taking a closer look at its performance, the most searched-for keyword Pages rank in the top 3 for is “was candy montgomery guilty” (600 MSV). The most difficult keyword it ranks in position one for is “when was the first bitcoin purchase” (KD: 76, MSV: 30). In other words, Pages still has a long way to go.

An n=1 (!) text similarity comparison with GoTranscript between Perplexity’s page for “bitcoin pizza day” and its four linked sources shows little evidence of plagiarism:

  1. nationaltoday.com/bitcoin-pizza-day/ (15% similarity).
  2. www.uledger.io/post/bitcoin-pizza-day-history (27% similarity).
  3. coinedition.com/bitcoin-pizza-day-a-700-million-reminder-of-cryptocurrencys-rise/ (15%).
  4. www.investopedia.com/news/bitcoin-pizza-day-celebrating-20-million-pizza-order/ (9%).
Text comparison between Perplexity’s and NationalToday’s article about Bitcoin Pizza Day (Image Credit: Kevin Indig)

The “missing” attribution issue seems to have been fixed, as the example below shows.

Perplexity highlights sources for answers at the top (Image Credit: Kevin Indig)

The results showed the chatbot at times closely paraphrasing WIRED stories, and at times summarizing stories inaccurately and with minimal attribution.

I wasn’t able to confirm or deny cases of hallucination, but I expect better models to get to a point at which they can summarize existing content flawlessly. The reality is, we’re not there yet. Google’s AI Overviews have also been shown to include wrong facts or make things up.

Google seems to have been able to improve the problem quickly, which is why I expect the degree of hallucination to drop.

One underlying issue of the plagiarism critique is that a search for the exact title of an article returns that article.

Of course, Perplexity should return a summary of an article when users prompt it. What else should Perplexity show? The same argument came up in the lawsuit between OpenAI and the NY Times.

Triggered

Besides the crawling issues Perplexity needs to fix, the media’s reaction seems to be triggered by Perplexity’s positioning.

One sentence in Perplexity’s announcement of Pages gets to the heart of the underlying issue:

“With Pages, you don’t have to be an expert writer to create high quality content.”

The page also mentions:

”Crafting content that resonates can be difficult. Pages is built for clarity, breaking down complex subjects into digestible pieces and serving everyone from educators to executives.”

All examples of Pages listed in the announcement are about “how to” or “what is” topics:

  • “Beginner’s guide to drumming”
  • “How to use an AeroPress”
  • “Writing Kubernetes CronJobs”
  • “Steve Jobs: Visionary CEO”
  • Etc.

That’s exactly the challenge AI poses to writers: AI can increasingly cover clearly defined content formats like guides or tutorials. I can see how this is triggering to journalists.

User-Directed Content

Note how Perplexity doesn’t create all the content for Pages but takes direction from humans through prompts (UDC).

Instead of writing a whole article, humans put the puzzle pieces together and their author bio stamp on a Page.

I expect the same to happen with other content types like reviews and platforms like Google, Tripadvisor, Yelp, G2 & Co. to provide corresponding tools to make content creation easier. The biggest challenge will be to keep quality high and reduce useless information to a minimum.

The big question is whether a build like Pages can compete with a purely human-written site like Wikipedia, which currently has 116,000 active contributors.

The bigger “Growth play” behind pages (IMHO) is how Perplexity creates AI (video) podcasts out of summarized articles that outrank original results.

“Perplexity then sent this knockoff story to its subscribers via a mobile push notification. It created an AI-generated podcast using the same (Forbes) reporting — without any credit to Forbes, and that became a YouTube video that outranks all Forbes content on this topic within Google search.”

Perplexity outranks publishers with video podcasts summarizing articles (Image Credit: Kevin Indig)

Google will have to figure out how to prevent LLMs from repurposing the content of publishers.

What remains after examining the facts is the realization of how difficult it is to balance giving an AI  answer while sending traffic to sources. Why should users click when most of their questions are answered?

On the other side of the coin, publishers themselves can provide summaries of their articles. Therefore, the key challenge for Perplexity – and anyone else who wants to create large-scale AI content for Search – is adding unique value on top of AI summaries.

The path to unique value from AI summaries and other AI content is personalization.

A system that can recognize your preferences of level of understanding for a topic can make AI summaries more useful to you. Perplexity is a wrapper around different LLMs, but if it collects significant information about users and personalizes output, it can add value beyond fast answers.

Device operating system makers like Alphabet and Apple have the biggest advantage when it comes to user data since they sit on top of the food chain.

A strong example is Apple Intelligence, which could likely answer questions currently provided by guides and tutorials on Google or Perplexity.

Apple Intelligence (abbreviated “AI” – nice one, Apple!) has full context through location (Apple Maps), third-party app usage, Siri prompts, email (Apple Mail), and other sources, which creates a nice base to personalize results on. The web is just one body of knowledge, with a much sexier one waiting on our Dropbox, Gmail inbox, and iPhone photos.

Today, personalized answers are a vision and a demo.

But at some point in the future, personalization will create better answers than any generic LLM summary and surely more than any human-written guide.

The value of defined and generic knowledge is on a collision course with LLM bombers. At the same time, the value of personalized knowledge, human experience, and trustworthy expert expertise is skyrocketing.


AI startup Perplexity wants to upend search business. News outlet Forbes says it’s ripping them off; Integrator vs Aggregator Growth

Perplexity AI Is Lying about Their User Agent

Perplexity CEO Aravind Srinivas responds to plagiarism and infringement accusations

What is Perplexity Pages?

Introducing Perplexity Pages

Wikipedia:About

Why Perplexity’s Cynical Theft Represents Everything That Could Go Wrong With AI


Featured Image: Paulo Bobita/Search Engine Journal

OpenAI’s Rockset Acquisition And How It May Impact Digital Marketing via @sejournal, @martinibuster

OpenAI acquired a technology from Rockset that will enable the creation of new products, real-time data analysis, and recommendation systems, possibly signaling a new phase for OpenAI that could change the face of search marketing in the near future.

What Is Rockset And Why It’s Important

Rockset describes its technology as a Hybrid Search, a type of multi-faceted approach to search (integrating vector search, text search and metadata filtering) to retrieve documents that can augment the generation process in RAG systems. RAG is a technique that combines search with generative AI that is intended to create more factually accurate and contextually relevant results. It’s a technology that plays a role in BING’s AI search and Google’s AI Overviews.

Rockset’s research paper about the Rockset Hybrid Search Architecture notes:

“All vector search is becoming hybrid search as it drives the most relevant, real-time application experiences. Hybrid search involves incorporating vector search and text
search as well as metadata filtering, all in a single query. Hybrid search is used in search, recommendations and retrieval augmented generation (RAG) applications.

…Rockset is designed and optimized to ingest data in real time, index different data types and run retrieval and ranking algorithms.”

What makes Rockset’s hybrid search important is that it allows the indexing and use of multiple data types (vectors, text, geospatial data about objects & events), including real-time data use. That powerful flexibility allows the technology to interact with different kinds of data that can be used for in-house and consumer-facing applications related to contextually relevant product recommendations, customer segmentation and analysis for targeted marketing campaigns, personalization, personalized content aggregation, location-based recommendations (restaurants, services, etc.) and in applications that increase user engagement (Rockset lists numerous case studies of how their technology is used).

OpenAI’s announcement explained:

“AI has the opportunity to transform how people and organizations leverage their own data. That’s why we’ve acquired Rockset, a leading real-time analytics database that provides world-class data indexing and querying capabilities.

Rockset enables users, developers, and enterprises to better leverage their own data and access real-time information as they use AI products and build more intelligent applications.

…Rockset’s infrastructure empowers companies to transform their data into actionable intelligence. We’re excited to bring these benefits to our customers…”

OpenAI’s announcement also explains that they intend to integrate Rockset’s technology into their own retrieval infrastructure.

At this point we know the transformative quality of hybrid search and the possibilities but OpenAI is at this point only offering general ideas of how this will translate into APIs and products that companies and individuals can create and use.

The official announcement of the acquisition from Rockset, penned by one of the cofounders, offered these clues:

“We are thrilled to join the OpenAI team and bring our technology and expertise to building safe and beneficial AGI.

…Advanced retrieval infrastructure like Rockset will make AI apps more powerful and useful. With this acquisition, what we’ve developed over the years will help make AI accessible to all in a safe and beneficial way.

Rockset will become part of OpenAI and power the retrieval infrastructure backing OpenAI’s product suite. We’ll be helping OpenAI solve the hard database problems that AI apps face at massive scale.”

What Exactly Does The Acquisition Mean?

Duane Forrester, formerly of Bing Search and Yext (LinkedIn profile), shared his thoughts:

“Sam Altman has stated openly a couple times that they’re not chasing Google. I get the impression he’s not really keen on being seen as a search engine. More like they want to redefine the meaning of the phrase “search engine”. Reinvent the category and outpace Google that way. And Rockset could be a useful piece in that approach.

Add in Apple is about to make “ChatGPT” a mainstream thing with consumers when they launch the updated Siri this Fall, and we could very easily see query starts migrate away from traditional search engine boxes. Started with TikTok/social, now moving to ai-assistants.”

Another approach, which could impact SEO, is that OpenAI could create a product based on an API that can be used by companies to power in-house and consumer facing applications. With that approach, OpenAI provides the infrastructure (like they currently do with ChatGPT and foundation models) and let the world innovate all over the place with OpenAI at the center (as it currently does) as the infrastructure.

I asked Duane about that scenario and he agreed but also remained open to an even wider range of possibilities:

“Absolutely, a definite possibility. As I’ve been approaching this topic, I’ve had to go up a level. Or conceptually switch my thinking. Search is, at its heart, information retrieval. So if I go down the IR path, how could one reinvent  “search” with today’s systems and structures that redefine how information retrieval happens?

This is also – it should be noted- a description for the next-gen advanced site search.  They could literally take over site search across a wide range of mid-to-enterprise level companies. It’s easily as advanced as the currently most advanced site-search systems. Likely more advanced if they launch it. So ultimately, this could herald a change to consumer search (IR) and site-search-based systems.

Expanding from that, apps, as they allude to.  So I can see their direction here.”

Deedy Das of Menlo Ventures (Poshmark, Roku, Uber) speculated on Twitter about how this acquisition may transform OpenAI:

“This is speculation but I imagine Rockset will power all their enterprise search offerings to compete with Glean and / or a consumer search offering to compete with Perplexity / Google. Permissioning capabilities of Rockset make me think more the former than latter”

Others on Twitter offered their take on how this will affect the future of AI:

“I doubt OpenAI will jump into the enterprise search fray. It’s just far too challenging and something that Microsoft and Google are best positioned to go after.

This is a play to accelerate agentic behaviors and make deep experts within the enterprise. You might argue it’s the same thing an enterprise search but taking an agent first approach is much more inline with the OpenAI mission.”

A Consequential Development For OpenAI And Beyond

The acquisition of Rockset may prove to be the foundation of one of the most consequential changes to how businesses use and deploy AI, which in turn, like many other technological developments, could also have an effect on the business of digital marketing.

Read how Rockset customers power recommendation systems, real-time personalization, real-time analytics, and other applications:

Featured Case Studies

Read the official Rockset announcement:

OpenAI Acquires Rockset

Read the official OpenAI announcement:

OpenAI acquires Rockset
Enhancing our retrieval infrastructure to make AI more helpful

Read the original Rockset research paper:

Rockset Hybrid Search Architecture (PDF)

Featured Image by Shutterstock/Iconic Bestiary