]*>(.*?)

The First-Ever UX Study Of Google’s AI Overviews: The Data We’ve All Been Waiting For via @sejournal, @Kevin_Indig

One thing I need you to understand about the groundbreaking data I’m about to show you is that no one has ever done this kind of analysis before.

Ever.

To our knowledge, no other independent usability study has explored a major web platform at this scale.

AI changes everything, and search is at the forefront.

Together with Eric van Buskirk and his team, I conducted a behavioral study that provides us with unique and mission-critical insights into how people use Google, especially AI Overviews (AIOs).

This data allows us all to better understand how people actually use the new feature and, therefore, better optimize for this new world of search.

We captured screen recordings + think-aloud sessions on 70 people (≈ 400 AIO encounters) to see what really happens when Google shows an AIO.

We tracked their scrolls, hovers, dwells, comments, and even their emotions!

The effort to gather and evaluate this data was high. It required:

• A solid five-figure USD investment.
• A team of six people.
• Combing through 13,500 words of annotations.
• Sifting through 29 hours of recordings.
• So many hours we lost count.

I want to call out that the study was directed by Eric Van Buskirk.

We designed the questions, focus points, and the method together, but Eric hired collaborators, ran the study, and delivered the results. Once the study was finished, we interpreted the data together.

Here’s a three-minute video summary of the results:

Boost your skills with Growth Memo’s weekly expert insights. Subscribe for free!

Executive Summary

Our usability study puts hard numbers behind what many SEO pros have sensed anecdotally:

• Traffic drain is real and measurable. Desktop outbound click-through rate (CTR) can fall by two-thirds the moment an AIO appears; mobile fares better, but still loses almost half its clicks.
• Attention stays up-screen. Seven in 10 searchers never read past the first third of an AIO; trust and visibility are won – or lost – inside a few lines.
• Demographics define user behavior. Younger mobile users embrace AI answers and social proof; older searchers still dig for blue links and authority sites. Query intents with high-risk outcomes (like Your Money or Your Life searches) also cause users to dig more into search for validation.
• The decision filter has changed. Brand/authority is now the first gate, search intent relevance the second; snippet wording only matters once trust is secured.
• Residual clicks follow community proof and video. Reddit threads, YouTube demos, and forum posts soak up roughly a third of the traffic that AIO leaves behind.

Together, these findings show that visibility, not raw referral traffic, is becoming the main currency of organic search.

Key Takeaways

Before you dig into the overall findings, here are the high notes:

1. AIOs kill clicks, especially on desktop: External click rates drop when an AIO block appears.
2. Most users skim only the top third of the panel: Citations or mentions for your brand must surface early to be seen. Median scroll = 30% of panel height; only a minority of users scroll past 75%.
3. Trust is earned through depth: Scroll-depth and stated trust move together (ρ = 0.38). Clear sources high up accelerate both trust and scroll-stop rate.
4. Age and device shape engagement: 25 to 34-year-olds on mobile are the power users: They pick AIO as the final answer in 1 of 2 queries.
5. Community and video matter post-AIO: When users do leave the SERP, many outbound clicks go to Reddit, YouTube, or forum posts – social proof seals decisions.

Methodology Summary

You’ll find a detailed methodology at the end of the article (and a methodology deep dive from Eric here), but here’s a short summary of how the data was collected:

We asked 70 U.S. searchers (42 on mobile, 27 on desktop) to complete eight real-world Google queries – six that trigger an AIO and two that do not – while UXtweak recorded their screens, scrolls, clicks, and think-aloud commentary.

Over 525 task videos (≈ 400 AIO encounters) were frame-by-frame coded by three analysts who logged scroll depth, dwell time, internal vs. external clicks, trust statements, and emotional reactions for every SERP element that held attention for at least five seconds.

The resulting 408 annotated results provide the quantitative spine – and the qualitative color – behind the findings you’re about to read.

We asked participants to complete these eight tasks:

1. Using Google Search, find a tax accountant in your area by searching as you typically would.
2. What are the best months to buy a new car?
3. Find a portable charger for phones under $15. Search as you typically would.
4. Find out how to transfer from PayPal to a bank.
5. Search Google for “promo code for a car rental.”
6. Search for two or three reasons why artificial sweeteners might cause health problems.
7. Search Google for “sell gift cards for an instant payment,” and imagine you have to choose one of the services mentioned.
8. Search Google for “how to waterproof fabric boots at home.”

1. How Do Users Actually Read AIO Content?

Several analyses have examined the impact of AIOs on click-through rates and organic traffic.

But no one has yet looked into how users actually engage with AIOs – until now.

In our analysis, we captured how far down the AIO users scroll, when they click the “show more” button, and where they dwell on the page.

Key Stats:

An overwhelming 88% of users clicked “show more” to expand truncated AIOs.

We measured how far down* the participants scrolled who looked at the AIO result for at least five seconds:

• Average scroll depth: 75%.
• Median scroll depth: 30%.

*0% = they never scrolled inside the box; 100% = they reached the very bottom at least once.

A few outliers skew the average.

The median is much more telling: Most users stop reading AIOs after the top third.

In total, 86% of participants “skimmed quickly,” meaning they didn’t take much time to read everything in the AIO but scavenged for key insights.

Dwell times averaged between 30-45 seconds, indicating meaningful user engagement rather than superficial interactions.

Eric, director of the study, found that 40% of questions end with statements like “I usually don’t go past this” or “AIO answers all my questions.”

The remainder, almost a third of the sessions, show people scanning AIOs, then choosing a brand site, video, Reddit thread, or .gov/.edu result instead. (“I like AIO, but I still prefer Reddit,” was a sentiment we heard.)

But who scrolls further down the AIO?

• Young people: Ages 25-34 years.
• Mobile users: An average of 54% mobile users vs. 29% desktop users keep scrolling the AIO.
• Searchers with an intent that reflects high stakes: Think tasks that involve financial or medical queries. Low-stakes searches, like coupon codes, are the opposite. Here’s a look at the average scroll depth across intents:
  • Health YMYL – 52%.
  • DIY or how-to – 54%.
  • Financial YMYL – 46%.
  • Decision timing (“best month to buy…”) – 41%.
  • Promo code queries – 34%.

When we asked participants about how much they trust AI-generated summaries, we got an average of 3.4 – quite high!

Why It Matters:

Similar to classic search results, aim to be cited as high up the AIO as possible to be the most visible.

When optimizing, we also need to consider the stakes of each individual, singular search query and what it might take for a person to verify a claim or find a trustworthy solution, whether the search is in YMYL topic or a general, traditionally low-risk topic.

This is more practical than the YMYL framework we’ve been using for a long time. The more a user has to lose when making the wrong decision, the more likely they are to engage deeply with AIOs.

Ultimately, our study shows that users engage more with an AIO out of skepticism. The higher the stakes are for a decision, the more they question the AIO. And the more they work to validate the AIO with sources outside of it.

Insight:

Users treat an AIO as a fact sheet: quick scan, expand if needed, minimal internal navigation.

You can see this in the difference between average and median scroll depth. Only a few users scrolled down to 75% of the AIO.

Users who end the task saying they trust the AIO are the same ones who have scrolled far enough to read citations or expanded paragraphs. Authoritative sources showing up high in the AIO accelerate trust.

Practical Takeaways:

1. Most people will never reach the bottom of the AIO, so valuable mentions and citations are only up high, similar to how classic search results work.
2. Similar to optimizing for Featured Snippets, when targeting AIOs, keep answers in content blocks concise, to the point, and simple.
3. Invest in your positioning, messaging, and becoming an authoritative source in your area of expertise. That way, users recognize your brand in the SERPs – and ideally before they search.

I’m dropping more insights and guidance on how to apply these learnings for paid subscribers later this week. Make sure you don’t miss it. Upgrade here.

2. What’s The Click-Through Behavior Like When AIOs Are Present?

AIOs give users answers before they click on web results.

Therefore, the logical question is how much less traffic can websites expect when AI Overviews show up?

Key Stats:

Everyone obviously wants to know how AIOs impact click-through rates. But clicks are just a proxy for completed user journeys.

While these are usually hard to track, we were able to figure out exactly when participants completed their journeys based on their commentary and screen tracking.

The remaining ~80% of queries were answered using:

• Organic and sponsored results.
• Community forums.
• Videos.
• Map packs.
• Other prominent SERP Features, like People Also Ask.

This observation actually fits Google’s narrative of AIOs being a “jumping-off point,” but I want to be clear that AIOs also kill a lot of clicks in the process, and our tasks require a higher level of skepticism than many highly searched queries.¹

Of course, there’s a difference between commercial and informational queries that we must keep in mind.

Notably, 4 out of 5 users progressed past the AIO, so ranking in the first organic or paid slots remains critical for monetizable queries.

Most answers (81 % on desktop and 78 % on mobile) for transactional and/or commercial queries came from other non-AIO SERP elements, such as:

• Organic links.
• Discussions and forums.
• Featured snippets.
• Promo‑code aggregators.
• Sponsored results.

But for AIO actions that took place in the Overview itself, here’s what we found:

• On mobile, 19% of participants clicked a citation-related element within the AIO panel, such as a link icon or hyperlinked text (excluding “show more” clicks).
• On desktop, users clicked internally within an AIO just 7.4% of the time.

Overall, our main AIO blocks contained few hyper-texted links, and on desktop, these links were nearly absent. The primary click out of the main panels was the (somewhat confusing) link icon.

The data we gathered from this part of the study confirms a few things:

• Don’t expect too much traffic, even when you’re cited high up the AIO. Traffic loss is inevitable and probably impossible to compensate for. (But there is hope.)
• Revenue models tied to sessions are suffering and will suffer more. (For example: Sites that rely on ads and affiliate models.)
• Marketing dashboards that track only visits are under-valuing visibility wins or hiding looming losses.
• The SERP battleground is shifting from rank to AIO presence. Budgets and optimization practices have to follow.

Insight:

The data shows that users treat AIO primarily as a read-only summary.

Users read, decide, and stay put.

Outbound traffic is the exception, not the rule. When AIOs are absent, outbound click rates rise to an average of 28% on desktop and 38% on mobile.

Notice how simple questions don’t require click-throughs in the following clip from the study:

And yet, there are cases in which organic results convince users to be better than AIOs:

Practical Takeaways:

Optimize content for AIO citations, but don’t measure referral traffic (ie, clicks) for success.

Instead, measure visibility by monitoring the following:

• Impressions (easy but fuzzy).
• Citation rank (how high up the AIO you’re cited).
• Share of Voice (how often you’re cited, how high, and how you show up in the organic results).

You also need to immediately communicate to leaders, stakeholders, partners, and clients that organic traffic is already or about to drop significantly.

For those who are subscribed to the paid version of The Growth Memo, we have a prepped slide deck to help you communicate these changes to your stakeholders coming out this week.

3. How Do People React To SERPs Emotionally?

Emotions drive decisions more than rationale.

As I wrote in Messy Middle:

“We’re more emotional animals and make more decisions from our gut than we like to admit.”

Besides engagement, we also wanted to know how users feel about the results they’re seeing.

Why? Because emotions have an impact on our decisions, from clicks to purchases.

Key Stats:

Why It Matters:

Emotion is tied to risk. Searchers are internally asking What’s at stake? When making a decision to trust a result

And as a result, high-stakes niches – or even expensive products – receive more skepticism and scrutiny from users.

This skepticism plays out in the form of clicks – a.k.a. your opportunities to convince people that you’re trustworthy.

The good news: Users don’t rely on AIOs only for YMYL queries; they also validate and verify with classic results.

In low-risk niches where the threat of picking a wrong answer is low – like coupon codes or certain informational queries – brands can focus on page speed and price.

Overall, here’s what stuck out the most from this segment of the study:

• Hesitation or confusion spikes on medical or money queries when AIOs cite unknown brands.
• Reassurance-seeking (opening a second organic link “to be sure”) appears in 38% of sessions where an AIO is present.
• No-reaction silent scans dominate product or local-intent tasks.

For high-stakes queries, users care about authoritative sources, as you can see in this clip from the study:

But organic results can still win if they signal better relevance.

1. Sites in the health and finance spaces have a higher chance of seeing lower traffic losses from AIOs.
2. Aim to get mentioned or linked from highly authoritative sources, like .gov sites.
3. Prioritize trust-building in your on-page experience to catch those double-check clicks. You can do this with visible editorial guidelines, expert authors + reviewers, and high-effort content production (original graphics, etc.).

4. What Influences The Type Of Result A User Chooses As Their Final Answer?

Up until now, my mental model of search – and I would argue the industry’s as well – was that users pick results by relevance: “Does it answer my question?”

But that has changed, and I think AI is a big reason.

We grouped over 550 think-aloud comments into four recurring themes to explain the new user behavior in the search results:

Source Trustworthiness (= Primary Click Motivator)

Whenever a recognised brand, authority site, .gov or .edu appeared, it was chosen first in 58% of the cases where such a link was present.

Comparison/Validation (= Secondary Driver)

After reading an AIO or Blue Link, 18% of users still opened a Reddit thread, YouTube video, or second organic result “just to double-check.”

Snippet/Preview Relevance (= Speeds Decision)

After clearing the trust gate, users scanned the two-line snippet, bolded query terms, or AIO phrasing. When the snippet looked off-topic, users skipped even trusted domains.

Top-Of-Page Visibility (= Skews The Decision)

Limited viewport and thumb ergonomics make “position-0” features (AIO, featured snippet) and rank-1 organic vastly more influential on phones.

First-screen links were chosen 71% of the time. Users only scroll when the topic feels risky.

Why It Matters:

It’s not just about matching the intent of the query anymore. The old notion of “search intent relevance only” is outdated.

Brand authority and trustworthiness compound: Once you’re trusted, you likely outrank unknown rivals – even without richer snippets.

Of course, placement matters, and SERP real estate above the fold is scarce … and skews user decisions.

Trust is the core ingredient when it comes to anything AI. Search is no exception.

Insight:

Users apply a rapid two-step filter that looks like this:

“Do I trust this result?” → “Does this result answer my question?”

Look at these two clips from the study and notice how the participant selects results he explicitly trusts:

You’ll hear the participant state:

• “Yelp is a good resource that I use a lot, so I’d probably click Yelp.”
• “I’ll try to find one that has decent reviews and that’s nearby.”
• “I trust Yelp.”

You’ll also hear this in Clip No. 2 above:

• “US News and World Reports is trustworthy. Edmonds is trustworthy.”
• “I picked this ’cause US News and World Reports is a trusted source, and they have a clear answer right here in the key takeaways.” (Note: They make information easy to find.)

Of course, there is nuance to this two-step filter.

The director of this study, Eric, adds the following observations:

1. How-to and evergreen intents (like waterproofing boots, selling gift cards, or coupon hunting) are easiest to satisfy for AIOs. Users feel the AI is “tried and tested” and “super helpful” for these stable facts.
2. Location-sensitive or personal-risk queries trigger more skepticism. One study participant shared aloud that, “It only says New York … that doesn’t help me,” and another shared, “I’d go straight to PayPal for accuracy.”
3. Medical-risk examples show a mixed approach: Some users praise the concise summary, others insist on cross-checking with authoritative sources like the Mayo Clinic or the NHS.

Ultimately, we’ve noticed that the more time users spend reading the AIO, the higher their chance of trusting the answer and being influenced by it.

This is a priming effect: Once a brand or concept appears in the AIO, it remains top-of-mind.

Practical Takeaways:

1. Trust is the gatekeeper. One of the biggest drivers of SEO success is how well you’ve earned “share of mind” before someone even sees your brand in an AIO or search result. Maybe that was always true, but AIOs make it non-negotiable now.
2. Being present in the AIO (ideally high up) is valuable because it leaves an impression on users. That’s where impressions as a metric become more valuable.

5. How Do Demographics Influence Search Behavior And Interactions With AIOs?

We often talk about user intent in SEO, but completely ignore demographics.

History has shown that technical jumps, like what we’re living through with AI right now, have a bigger impact on younger demographics.

The same is true of Search.

Our research found stark differences in how people of different ages engage with the search results.

Below, you’ll see the percentage of time when an organic result was chosen, whether there was an AIO present for the user’s review or not.

Why It Matters:

One-size-fits-all SEO practices don’t work anymore.

Just like for other social or content platforms, segmentation by demographic becomes as critical as segmentation by keyword intent.

Insight:

AIO adoption is generational.

Older audiences are still relying heavily on classic organic results. Younger demographics are more likely to focus on the AIO and validate with Reddit.

Practical Takeaways:

Prioritize content and SERP Feature bets by age segment.

For brands that target an older audience, double down on classic organic search. Don’t over-index on AIOs. The exception here would be queries with a local intent or online shopping searches.

In these cases, user intent overpowers age preferences.

Quick reminder here: Premium subscribers get expanded info later this week. Upgrade to paid.

6. How Do Devices Impact User Behavior?

Devices reflect search context.

Mobile devices are used more often on the go, which is why mobile searches are more likely to have local intent and SERP Features like local packs.

Mobile users are also more restrained in their behavior due to smaller screen real estate. These factors are also reflected in how users engage with AIOs.

Why It Matters:

You need to ensure mobile snippets and structured data are flawless; they get more scrutiny.

Mobile is now the primary remaining source of incremental Google traffic. Optimize for it first.

Insight:

Vertical scrolling and thumb ergonomics make mobile users dig deeper and click out more.

And when an AIO is missing, users revert to classic “blue link” behavior, especially on mobile, where more than one-third of searches produced a click to a non-Google site.

Practical Takeaways:

You need to track and compare mobile and desktop SERPs.
We missed this in classic SEO, and now it’s so much more important.

To prioritize which format you optimize for, you must validate that you get more mobile users to your site first (use Google Search Console).

If mobile is important to your target audience, regularly run separate mobile rank and snippet audits. And optimize the above-the-fold experience, in addition to:

• Making the page skimmable.
• Shortening time-to-value on the page (essentially, the time it takes to resolve the query or reach an insight from your site).
• Simplifying navigation on the page and site.

7. How And When Do Users Engage With Community-Based, Video-Based, And Shopping Carousel Content?

The controversial rise of Reddit often leaves us wondering why Google gives community content so much prominence across all topics and verticals.

Our study explains what users really do.

Key Stats:

We looked at where clicks go when users leave Google or want to validate answers:

Keep in mind that SERP Features and corresponding user behavior vary by question or task performed.

In this study, only one task surfaced video results: “how to waterproof fabric boots at home.”

And here’s how users in this study responded to video results:

• Users watched the preview frames, hovered for autoplay, then clicked through to YouTube in 5 of the 7 cases.
• Although videos made up less than 2% of all logged elements, their 37-second dwell time exceeds AIO dwell time itself (31 seconds).
• Users linger to watch autoplay previews or scroll thumbnails before deciding to click through.

For shopping-related tasks, we noticed the following:

• 30% of clicks go to local packs.
• 26.4% of clicks go to shopping modules (product grids).
• 13.2% of clicks go to text ads.
• 40% of clicks went to paid-organic results (text + PLAs).
• 7 out of 10 clicks bypassed classic organic links in favour of Google‑curated verticals or ads.

By the way, Amazon was a huge competitor to the shopping carousel.

Many people said, “I would just go to Amazon” (see clip below):

This study participant states: “Typically, I go to Amazon … scroll past the sponsored results and look for something with a lot of reviews.”

Why It Matters:

Social proof platforms (Reddit, YouTube) absorb the demand that AIOs can’t satisfy. Be present there.

Insight:

Community proof-points matter. When users leave the SERP after looking at an AIO, community links receive a lot of those clicks (18% when AIOs are not present).

People – especially the younger cohort that trusts AIO the most – use forums to get a (validating) voice from another human. Users in their 20s to 30s clicked Reddit or YouTube far more than older cohorts.

For some queries, like how-tos, users skip the AIO intentionally because they expect richer media, like videos.

Practical Takeaways:

1. Invest in Organic Reddit (or the most relevant forum in your industry) when and if it appears for your most relevant queries. Seek both citations and social proof, as they reinforce each other.
2. Optimize video thumbnails and the first 15 seconds. Users decide whether or not to click from the autoplay preview; if the opening doesn’t show the task in action, they skip.

Conclusion: Welcome To The New World Of Search

You made it to the end! Congratulations to you and your attention span (or did you just scroll here 🤔?).

To summarize everything you just (hopefully) read: If your brand isn’t surfaced in the first third of an AIO, it’s effectively invisible.

Search has flipped from a click economy to a visibility economy.

And within that economy, the new currency is authority, which now outranks search intent relevance.

Users ask, “Do I trust this brand?” before they even consider the answer.

If I had to boil the findings down to one sentence, it would be this: Users treat an AIO as a fact sheet: They quickly scan, expand if needed, and use minimal internal navigation.

Top Takeaways For Operators:

1. Shift KPIs from clicks to presence. Track how often, how high, and for which queries your brand appears in AIO.
2. Lead with authority. Invest in expert endorsements, .gov/.edu links, and PR that earns immediate trust.
3. Package answers for skimmers. Key-fact boxes, bullets, and schema matter more than ever.
4. Own the validation click. Seed Reddit threads, video demos, and comparison guides – users still seek a second opinion.
5. Segregate desktop and mobile strategy. Treat desktop as a branding surface; fight for mobile if you need traffic.

Top Takeaways For Decision Makers:

1. Expect – and budget for – a structural drop in organic sessions. AIOs cut outbound clicks roughly in half on desktop and by a third on mobile; revenue models tied to sessions (ads, affiliate) need hedging strategies.
2. Shift KPIs and tooling from “rank” to “share of voice in AIO.” Track how often, how high, and for which queries your brand appears in the panel; classic position-tracking alone masks looming losses. Keep in mind we’re still refining the new metrics model.
3. Invest in authority signals that secure trust instantly. Recognition by .gov, .edu, expert reviewers, or high-profile PR sways 58% of users to choose a cited source first. Brand trust precedes relevance in the new decision filter.
4. Allocate resources to validation channels – Reddit, YouTube, forums – where many residual clicks go after an AIO. Owning the follow-up click preserves influence even when Google keeps the first.

Open Questions That Still Matter

• Citation mechanics. How does Google choose which sources surface in the collapsed AIO, and in what order?
• Attribution leakage. Will Search Console or GA ever expose AIO-driven impressions so brands can value “on-SERP” exposure?
• Monetization models. If outbound traffic keeps shrinking, how will publishers, affiliates, and SaaS products replace lost session-based revenue?
• Personalization vs. authority. Will future AIOs weigh personal history over global trust signals – and can brands influence that balance?
• Regulatory impact. Could antitrust or copyright actions force Google to show more outbound links  – or fewer?
• Behavior over time. Do users acclimate to AIOs and eventually click less (or more) as trust grows?

Hint: Paid subscribers can get answers to these questions (and can send me any question that’s top of mind!) related to this study.

Additional Resources

Other primary research that puts the qualitative data into perspective:

Methodology

Study Design And Objective

We conducted a mixed-methods, within-subjects usability study to quantify how Google’s AI Overviews (AIO) change user behavior.

Each participant completed eight live Google searches: six queries that consistently triggered an AIO and two that did not. This arrangement lets us isolate the incremental effect of AIO while holding person-level variables constant.

Participants And Recruitment

Sixty-nine English-speaking U.S. adults were recruited on Prolific between 22 March and 8 April 2025.

Eligibility required a ≥ 95% Prolific approval rate, a Chromium-based browser (for the recording extension), and a functioning microphone.

Participants chose their own device; 42 used mobile (61%) and 27 used desktop (39%).

Age distribution was: 18-24 yrs 29%, 25-34 yrs 30%, 35-44 yrs 12%, 45-54 yrs 17%, 55-64 yrs 3%, 65+ yrs 3%.

A pilot with eight users refined instructions; 18 further sessions were excluded for technical failure and four for non-compliance. The final dataset contains 525 valid task videos.

Task Protocol

Each session ran in UXtweak’s Remote Moderated mode.

After reading a task prompt, the participant navigated to google.com, searched, and spoke thoughts aloud. They declared a final answer (“I’m selecting this because…”) before clicking “Done” in an overlay.

Task set:

1. Local service (“find a tax accountant near you”) – no AIO.
2. Decision timing (“best month to buy a car”) – AIO.
3. Low-cost product (“portable charger < $15”) – no AIO.
4. Transactional YMYL (“transfer PayPal to bank”) – AIO.
5. Coupon/deal (“car-rental promo code”) – AIO.
6. Health YMYL (“why artificial sweeteners might cause health problems”) – AIO.
7. Finance YMYL (“sell gift cards for instant payment”) – AIO.
8. DIY how-to (“how to waterproof fabric boots”) – AIO.

Capture Stack

UXtweak recorded full-screen video (1080p desktop or device resolution mobile), cursor paths, scroll events, and audio. Recordings averaged 25 min; incentives were $8 USD.

Annotation Procedure

Three trained coders reviewed every video in parallel and logged one row per SERP element that held attention ≈ for 5 seconds or longer. Twenty-three variables were captured, grouped as:

• Structural – participant-ID, task-ID, device, query.
• Feature – element type (AIO, organic link, map pack, sponsored, video pack, shopping carousel, forum, etc.).
• Engagement – scroll depth inside AIO (0/25/50/75/100%), number of scroll gestures, dwell-time (s), internal clicks, outbound clicks.
• Behavioral – spoken reaction (hesitation, confusion, reassurance-seeking, none), reading style (skim, re-read, etc.), AIO button used (show-more, citation click, carousel chip).
• Outcome – final answer, satisfaction flag, explicit trust flag.

The research director (Eric van Buskirk) spot-checked 10% of videos. Inter-coder agreement: dwell-time SD ± 3 s; Cohen’s κ on trust category = 0.79 (substantial).

Data Processing And Metrics

Annotations were exported to Python/pandas 2.2. Scroll values entered as whole numbers were normalised to fractions (e.g., 80 → 0.80).

The 99th percentile of dwell was Winsorised to dampen outliers. This produced 408 evaluated SERP elements and ≈ 350 valid AIO observations.

Statistical Analysis

Descriptives (means, medians, proportions) were stratified by device, age, and query intent.

Spearman rank correlations tested monotonic relationships among scroll %, dwell, trust, and query-refinement counts (power >.8 to detect ρ ≥ .25).

Welch t-tests compared mobile vs desktop means; McNemar χ² compared click-through incidence with vs without AIO.

Reliability And Power

With n ≈ 350 AIO rows, the standard error for a proportion of .50 is ≈ .05; correlations ≥ .30 are significant at α =.05. Cross-coder checks ensured temporal metrics and categorical judgements were consistent.

Limitations

Sample skews young (58% ≤ 34 yrs) and U.S.-based; think-aloud may lengthen dwell by ~5-10 s. Coder-judged trust/emotion involves subjectivity despite reliability checks.

Study window overlaps Google’s March 2025 core update; SERP UI was in flux. Findings generalise to Chromium browsers; Safari/Firefox users were not sampled.

Ethical Compliance

Participants gave informed consent; recordings stored encrypted; no personally identifying data retained. Study conforms to Prolific’s ethics policy and UXtweak TOS.

This narrative supplies sufficient procedural and statistical detail for replication or secondary analysis.

1 AI Overviews: About last week 

---

SEJ’s Content & SEO Strategist Shelley Walsh prerecorded an interview with Kevin before the launch to talk about his research. For more explanation about his findings, watch below.

More Resources:

---

Featured Image: Paulo Bobita/Search Engine Journal

Google Ads AI Vs. Third-Party AI Tools: Comparison For Google Ads Creatives

Every day, marketing teams face a crucial decision: Should they rely on Google’s built-in AI tools or invest in custom solutions for specific ad campaign tasks?

I’ve watched this debate play out countless times with clients.

Google continues adding more AI features for tasks like ad copy generation, headline creation, image generation, and product feed optimization.

Meanwhile, specialized tools and custom solutions are thriving, and no real breakthrough for Google AI can be seen.

Recent research supports this tension.

Gherheș et al. (2025) found that while AI-generated content can outperform human-created alternatives in certain contexts, the quality varies significantly depending on implementation and purpose.

Their study revealed that over 50% of users preferred AI-generated informative content over sensationalized approaches, suggesting that how AI is deployed matters more than the technology itself.

But which approach actually delivers better results? And at what cost?

As Pavlik (2024) notes in his analysis of AI in journalism, tools like ChatGPT don’t simply replace human creativity but rather present opportunities for “improving the quality and effectiveness” of creative work when properly integrated into existing workflows.

A recent study by Ameet Khabra compared the performance of human-written versus AI-generated ad copy in Google Ads campaigns.

Over an eight-week period with a $500 budget, human-crafted ads significantly outperformed AI-created content from Copy AI, achieving 60% more clicks, a 1.33% higher click-through rate, and a lower cost per click ($4.85 vs. $6.05).

Researchers attributed human copywriters’ superior performance to their ability to understand audience emotions, employ creativity and emotional appeal, adapt to specific contexts, and leverage cultural nuances that AI still struggles to replicate.

While acknowledging AI’s evolving capabilities and potential value as a supplementary tool, the study emphasizes the enduring importance of human creativity in crafting compelling advertising messages that drive engagement and conversions.

Regardless of these mixed research findings, one thing is certain: AI is increasingly embedded in creative processes across marketing, and its integration is inevitable.

The question isn’t whether AI will play a role in advertising creation, but rather how marketers can best leverage these tools to enhance their campaigns.

As AI capabilities evolve rapidly, today’s limitations may be tomorrow’s strengths. With this inevitability in mind, marketers need practical guidance on navigating the current landscape of available solutions.

This article compares Google Ads integrated AI tools against third-party and custom solutions for creative and optimization tasks specifically.

AI-Generated Ad Copy

Google AI Automatically Created Assets

Google’s AI text generator aims to streamline the ad creation process by converting basic product descriptions into campaign-ready assets.

The platform encourages advertisers to input unique selling propositions and key product features to generate contextually relevant ad copy.

Upon testing this tool with a simulated video game business specializing in refurbished PlayStation 5 consoles and games, the performance fell notably short of expectations.

The output quality was inconsistent, but more concerning were the significant compliance issues observed.

In one particularly problematic instance, the system generated the phrase “Welcome to the Amazon® Website” as suggested ad text, presenting a clear trademark infringement risk and potential legal exposure for advertisers.

Such critical errors highlight a fundamental limitation in Google’s native AI solution: While offering workflow convenience, it demonstrates inadequate safeguards for brand compliance and legal protection.

The system also produced contextually inappropriate messaging, such as “PlayStation 5 Problems Solved,” which misaligned with sales-oriented campaign objectives by suggesting repair services or technical support rather than product offerings.

Without careful human review, these problems make the tool risky to use, especially for businesses in competitive markets where mistaken identity or inaccurate representations could lead to serious legal issues and damage to your reputation.

When generating longer headlines, there were much fewer results.

Only three ad suggestions appeared, one of which included free shipping information for orders over $50, which was a hallucination, as this information was never disclosed in the prompt or the landing page.

Creating descriptions was even worse, as there was only one ad suggestion and not even a good one from a copywriting perspective.

After trying with different prompts, I was able to get at least five new descriptions out of Google AI.

Still, the results were quite disappointing. The ad copy contained hallucinations like the “free shipping over 100 USD,” as well as the business name “Example Video Games,” instead of using the business name of the account or extracting it from the landing page or URL.

Overall, underwhelming results, considering Google is one of the biggest companies on earth and owns the biggest online advertising platform.

Third-Party Ad Copy Creation

While Google’s AI text generator struggles with brand accuracy and contextual relevance, several general-purpose AI models offer more sophisticated ad copy creation capabilities that balance automation with quality control.

Leading general AI assistants like Claude, ChatGPT, and Gemini represent compelling alternatives for marketers seeking higher-quality ad copy generation.

Unlike Google’s more constrained system, these platforms offer greater flexibility in handling nuanced prompting and brand-specific requirements.

In testing with our video game business scenario, we prompted each model to create headlines for refurbished PlayStation 5 consoles.

The results demonstrated significant advantages over Google’s native offering:

• Claude 3.7 produced premium-positioned headlines like “Save On Certified PS5 Consoles,” “Quality PS5 | Full Warranty,” and “Premium PS5 | Fast Shipping” that emphasize both value and quality assurance. Claude’s headlines maintained strong brand positioning while highlighting availability (“PS5 Consoles In Stock Now”) and price advantages (“PS5 Consoles 30% Off Retail”) without sacrificing perceived value.
• ChatGPT (o3-mini) focused more on emotional appeal and deal framing with options such as “PS5 Deals You’ll Love,” “Get More, Spend Less PS5,” and “Budget PS5, Premium Fun.” ChatGPT’s approach effectively balanced affordability messaging with aspirational elements, potentially appealing to both value-conscious and experience-focused consumers.
• Gemini 2.0 took a more direct value-oriented approach with straightforward headlines like “Refurbished PS5 Deals,” “Cheap Used PS5,” and “Discount PS5 Titles.” While less nuanced in positioning, Gemini’s headlines clearly communicate the core offering and may perform well for price-sensitive segments or direct response campaigns.

All three models demonstrated superior context awareness compared to Google’s native tool, with each showcasing different strategic approaches to headline creation.

They successfully avoided the hallucinations and brand confusion issues observed in Google’s Ad tool, while providing greater headline variety tailored to different marketing objectives.

The key advantage these general AI assistants offer is their adaptability and more refined understanding of marketing language.

By providing detailed prompting with brand guidelines, target audience information, and specific messaging requirements, marketers can achieve significantly better results than with Google’s more limited integrated tool.

For businesses where ad copy directly impacts conversion rates, leveraging these more sophisticated AI options can yield higher-quality creative assets that better represent brand positioning and speak more effectively to customer needs.

Despite Gemini’s relevant headline ideas, it struggled to adhere to the 30-character limit for some prompts on Google Ads headlines – a surprising limitation given that Gemini is Google’s own AI model and would be expected to understand Google Ads guidelines inherently – while Claude and ChatGPT consistently produced properly sized headlines without major additional editing or truncation.

Image Generation

Google AI Image Generation

Image generation is another area where AI can really shine and reduce the workload.

Images are a core asset in ecommerce, not only used for product images, but also for category pages, shop banners, display ads, and more.

For our virtual video game business, I tried to create some images matching our PlayStation 5 asset group. The results were interesting to say.

The first created image looks very similar to an Xbox. Specifically, an Xbox One S or an Xbox Series S, which is the latest model.

Now, there are no logos or trademarks seen, and the form factor is a little off.

Even more interesting, depending on the exact prompt, Google AI shows an error message related to branded items and content restrictions.

Another image created looks a little more like a PlayStation, but not as described and advertised as a PlayStation 5, but rather an older PlayStation 4 model.

Again, the content restrictions are most likely responsible for the results.

While the image results are somewhat disappointing for those branded items, it is understandable that Google AI follows content restrictions and brand guidelines to avoid any legal issues, as the PlayStation is a trademark of Sony, and the Xbox is a trademark of Microsoft.

It’s interesting to see that Google AI tries to work around this limitation and still creates an image, but in that specific case, the image is more or less useless, as there is little value in showcasing a non-existent video game console.

A question here would be why the content restrictions and guidelines did not apply to text creation when the text asset “Welcome to the Amazon® Website” was created.

To check the image creation quality, I tried a different approach for non-branded items in the dog food category.

The image is good at first glance since multiple products are shown with a dog in the picture, supporting the category, but some things are off.

The text in the image is still a mess for Google AI. Plus, the proportions are wrong. The dog is way too small, considering the cans of dog food displayed, which are small items.

Better than video game consoles, but still not good enough to rely solely on Google AI without any backup or alternative.

Third-Party Image Generation

ChatGPT

Using the same prompts to create images, ChatGPT delivers amazing results compared to the Google Ads integrated image creator.

Visually, it was able to recreate a PlayStation video game console with a gaming controller.

ChatGPT even got details right, except the brand logo, which might be due to some brand protection measures.

Also, the latest Xbox model was created with in-depth details.

This time, even the Xbox logo was created, maybe because ChatGPT and Microsoft have made a trademark deal of some sort? Or trademark restrictions have some loopholes.

At last, the creation of the dog food image was also a success.

The prompt included the brand to be named “Doug’s Best Dog Food,” which was perfectly written on the product, along with a nicely placed bowl full of pellets in front of a golden retriever.

In comparison, Google AI was able to create a decent image, but upon closer look, issues with displaying words were apparent, which ChatGPT could handle perfectly.

Qwen

Qwen is an image generation tool based on Deepseek, which is a Chinese-based AI developer.

The image from Qwen clearly had an “AI” look compared to the ChatGPT or Google AI image.

However, it got the brand name “Doug’s Best Dog Food” right. With some improvements, Qwen can produce decent images, if you are okay with having a digital image look.

Google AI was able to create a more real-life looking image, with the downside of not being able to display the brand name correctly.

Video Creation Tool

Google Ads Video Creation Tool

Google’s built-in video creation tool aims to make video advertising accessible to marketers without production resources.

The tool covers multiple marketing objectives – from brand awareness and consideration to direct sales and app promotion – accommodating various business categories, including apps, products, and services.

It offers flexibility with vertical, square, and horizontal formats in lengths ranging from quick six-second spots to 15-second and longer videos.

However, the tool’s output quality reveals its limitations.

Most videos are essentially slideshows, stitching together static images, logos, and text overlays rather than fluid video content.

While this approach democratizes video ad creation, the results often lack the polish and engagement power of properly produced video content.

For many marketers, this represents the fundamental tradeoff of Google’s native tools: accessibility and integration vs. creative limitations that might impact performance.

At best, marketers get a nice-looking slideshow; speaking of a serious advertising video would be too much.

The better templates are mostly for app-related videos, where at least some kind of animation is included with a finger doing some phone touch gestures.

Overall, the native video creation tool serves as a backup for marketers who need a video immediately and don’t have any tools on hand.

In any other case, it’s best to postpone video creation and start with a more decent tool.

Third-Party Video Creation

Canva Video Creation

Canva makes much better videos than Google Ads’ built-in tools with almost the same effort.

Google mostly creates basic slideshows, but Canva gives you thousands of professional templates, animations, and stock videos to use in your marketing.

The simple drag-and-drop design lets you make engaging videos with smooth transitions and text effects that keep viewers engaged.

Unlike Google’s static slideshows, Canva creates flowing video content that looks professionally made.

If you spend just a few more minutes using Canva instead of Google’s tool, your videos will look much more professional and likely perform better with your audience.

Qwen

Alibaba’s Qwen is a strong competitor to Google Ads’ basic video tools, giving marketers better videos without needing special skills.

While Google just makes simple slideshows, Qwen uses AI to turn your images and text into dynamic videos with smooth movements and professional transitions.

The tool is great at automatically creating cohesive visual stories even from minimal input, adding motion to still images in ways that look professional.

What stands out is how Qwen creates animations that actually match your product type and brand style, avoiding the one-size-fits-all look of Google’s templates.

Though not as well-known as Canva in the West, Qwen’s AI approach produces polished videos that look intentionally designed rather than template-made, making it a great choice for marketers who want better videos than what Google offers.

For the example of a dog food brand, Qwen delivered exceptional results.

With the first prompt, Qwen created a five-second clip of golden retrievers playing around and going to a human hand to eat dog food from the hand.

Not only did the video look pretty close to real life compared to the Qwen image generation “AI look,” but Qwen also did this as a free tool. No cost involved.

If you compare this to the Google video creation, which is basically a PowerPoint presentation, Qwen makes a really good performance.

Sora

Another great video tool is Sora from OpenAI.

Since Sora is included in the $20 Premium membership, you can generate videos at almost no cost, though with some limitations on video quality and length.

Still, there are a few tools out there that can generate decent AI video output for that cost.

Product Image Improvements

Product Studio

The Product Studio for Merchant Center Next is a beta image optimization tool within the Merchant Center, also accessible via the Google App within Shopify.

It allows for creating product images in various scenes, as well as removing backgrounds and increasing image quality.

These are two tries to display a gaming controller in a scene.

Although the quality of the product image has remained reasonably good, the scenes are barely usable.

The image processing prompt was “Showcase this controller in a living room, in front of a TV with neon lighting.”

In practice, the desired scene was not even remotely depicted. The controller in front of notebooks or pens is out of place; the second attempt resulted in three black backgrounds and a fiery background.

Free Alternatives To Google’s Product Studio

Unlike Google’s Product Studio, which struggles with accurate scene generation as shown in the gaming controller example, several free tools offer more reliable image optimization capabilities.

Canva’s free tier includes a background removal tool that produces clean cutouts with remarkable accuracy.

While scene creation is more limited in the free version, you can still place products on various pre-designed backgrounds or use their extensive template library to create more contextually appropriate product displays than what you experienced with Google’s tool.

To remove backgrounds, use remove.bg, which is a specialized tool that focuses exclusively on background removal with impressive results, even for complex products like your gaming controller.

The free version has size limitations but delivers professional-quality cutouts that can then be placed into scenes using other tools.

For everything more complex, GIMP is a free and capable tool. This open-source image editor provides robust capabilities for both background removal and scene composition.

Though it has a steeper learning curve, GIMP offers precise control over image quality enhancement and realistic product placement.

Final Thoughts

Google’s native AI tools, while conveniently integrated into their advertising platform, consistently underperform compared to third-party alternatives.

The evidence is clear and concerning. Google’s AI ad copy generator produced legally problematic content with brand infringement risks and hallucinated product details.

Its image generation produced visually inaccurate representations. The video creation tool delivered little more than basic slideshows rather than engaging video content.

Meanwhile, third-party solutions or Google’s own Gemini model used externally demonstrated superior capabilities across all creative functions:

• General-purpose AI assistants like Claude and ChatGPT produced more compelling, accurate, and compliant ad copy.
• Specialized tools like Canva, Remove.bg, and Photopea offered vastly superior image manipulation options.
• Video creation platforms like Canva and Qwen delivered professional-quality animation and transitions impossible with Google’s basic tools.

This performance gap reveals the fundamental tradeoff marketers face: convenience of integration vs. creative quality and performance.

Google’s in-platform AI tools provide workflow efficiency but at the significant cost of creative limitations, brand safety concerns, and potential legal exposure.

For marketers serious about campaign performance, the path forward is clear: Leverage external AI solutions for creative development, then import these higher-quality assets into Google’s advertising platform.

This hybrid approach maintains the advantage of Google’s targeting and delivery mechanisms while avoiding the substantial limitations of their creative AI tools.

As AI continues to evolve in marketing, successful advertisers will be those who strategically select the right tools for each specific function rather than defaulting to in-platform options for convenience alone.

The evidence suggests that, for now, the marketing advantage lies decidedly with those willing to look beyond Google’s native AI for their creative development needs.

More Resources:

---

Featured Image: KinoMasterskaya/Shutterstock