Cloudflare Report: Googlebot Tops AI Crawler Traffic via @sejournal, @MattGSouthern

Cloudflare published its sixth annual Year in Review, offering a comprehensive looks at Internet traffic, security, and AI crawler activity across 2025.

The report draws on data from Cloudflare’s network, which spans more than 330 cities across 125 countries and handles over 81 million HTTP requests per second on average.

The AI crawler findings stand out. Googlebot crawled far more web pages than any other AI bot, reflecting Google’s dual-purpose approach to crawling for both search indexing and AI training.

Googlebot Top AI Crawler Traffic

Cloudflare analyzed successful requests for HTML content from leading AI crawlers during October and November 2025. The results showed Googlebot reached 11.6% of unique web pages in the sample.

That’s more than 3 times the pages seen by OpenAI’s GPTBot at 3.6%. It’s nearly 200 times more than PerplexityBot, which crawled just 0.06% of pages.

Bingbot came in third at 2.6%, followed by Meta-ExternalAgent and ClaudeBot at 2.4% each.

The report noted that because Googlebot crawls for both search indexing and AI model training, web publishers face a difficult choice. Blocking Googlebot’s AI training means risking search discoverability.

Cloudflare wrote:

“Because Googlebot is used to crawl content for both search indexing and AI model training, and because of Google’s long-established dominance in search, Web site operators are essentially unable to block Googlebot’s AI training without risking search discoverability.”

AI Bots Now Account For 4.2% of HTML Requests

Throughout 2025, AI bots (excluding Googlebot) averaged 4.2% of HTML requests across Cloudflare’s customer base. The share fluctuated between 2.4% in early April and 6.4% in late June.

Googlebot alone accounted for 4.5% of HTML requests, slightly more than all other AI bots combined.

The share of human-generated HTML traffic started 2025 at seven percentage points below non-AI bot traffic. By September, human traffic began exceeding non-AI bot traffic on some days. As of December 2, humans generated 47% of HTML requests while non-AI bots generated 44%.

Crawl-to-Refer Ratios Show Wide Variation

Cloudflare tracks how often AI and search platforms send traffic to sites relative to how often they crawl. A high ratio means heavy crawling without sending users back to source sites.

Anthropic had the highest ratios among AI platforms, ranging from approximately 25,000:1 to 100,000:1 during the second half of the year after stabilizing from earlier volatility.

OpenAI’s ratios reached as high as 3,700:1 in March. Perplexity maintained the lowest ratios among leading AI platforms, generally below 400:1 and under 200:1 from September onward.

For comparison, Google’s search crawl-to-refer ratio stayed much lower, generally between 3:1 and 30:1 throughout the year.

User-Action Crawling Grew Over 20X

Not all AI crawling is for model training. “User action” crawling occurs when bots visit sites in response to user questions posed to chatbots.

This category saw the fastest growth in 2025. User-action crawling volume increased more than 15 times from January through early December. The trend closely matched the traffic pattern for OpenAI’s ChatGPT-User bot, which visits pages when users ask ChatGPT questions.

The growth showed a weekly usage pattern starting in mid-February, suggesting increased use in schools and workplaces. Activity dropped during June through August when students were on break and professionals took vacations.

AI Crawlers Most Blocked In Robots.txt

Cloudflare analyzed robots.txt files across nearly 3,900 of the top 10,000 domains. AI crawlers were the most frequently blocked user agents.

GPTBot, ClaudeBot, and CCBot had the highest number of full disallow directives. These directives tell crawlers to stay away from entire sites.

Googlebot and Bingbot showed a different pattern. Their disallow directives leaned heavily toward partial blocks, likely focused on login endpoints and non-content areas rather than full site blocking.

Civil Society Became Most-Attacked Sector

For the first time, organizations in the “People and Society” vertical were the most targeted by attacks. This category includes religious institutions, nonprofits, civic organizations, and libraries.

The sector received 4.4% of global mitigated traffic, up from under 2% at the start of the year. Attack share jumped to over 17% in late March and peaked at 23.2% in early July.

Many of these organizations are protected by Cloudflare’s Project Galileo.

Gambling and games, the most-attacked vertical in 2024, saw its share drop by more than half to 2.6%.

Other Key Findings

Cloudflare’s report included several additional findings across traffic, security, and connectivity.

Global Internet traffic grew 19% year-over-year. Growth stayed relatively flat through mid-April, then accelerated after mid-August.

Post-quantum encryption now secures 52% of human traffic to Cloudflare, nearly double the 29% share at the start of the year.

ChatGPT remained the top generative AI service globally. Google Gemini, Windsurf AI, Grok/xAI, and DeepSeek were new entrants to the top 10.

Starlink traffic doubled in 2025, with service launching in more than 20 new countries.

Nearly half of the 174 major Internet outages observed globally were caused by government-directed shutdowns. Cable cut outages dropped nearly 50%, while power failure outages doubled.

European countries dominated Internet quality metrics. Spain topped the list for overall Internet quality, with average download speeds above 300 Mbps.

Why This Matters

The AI crawler data should affects how you think about bot access and traffic.

Google’s dual-purpose crawler creates a competitive advantage. You can block other AI crawlers while keeping Googlebot access for search visibility, but you can’t separate Google’s search crawling from its AI training crawling.

The crawl-to-refer ratios help quantify what publishers already suspected. AI platforms crawl heavily but send little traffic back. The gap between crawling and referring varies widely by platform.

The civil society attack data matters if you work with nonprofits or advocacy organizations. These groups now face the highest rate of attacks.

Looking Ahead

Cloudflare expects AI metrics to change as the space continues to evolve. The company added several new AI-related datasets to this year’s report that weren’t available in previous editions.

The crawl-to-refer ratios may change as AI platforms adjust their search features and referral behavior. OpenAI’s ratios already showed some decline through the year as ChatGPT search usage grew.

For robots.txt management, the data shows most publishers are choosing partial blocks for major search crawlers while fully blocking AI-only crawlers. The year-end state of these directives provides a baseline for tracking how publisher policies evolve in 2026.


Featured Image: Mamun_Sheikh/Shutterstock

Google Updates Search Live With Gemini Model Upgrade via @sejournal, @martinibuster

Google has updated Search Live with Gemini 2.5 Flash Native Audio, upgrading how voice functions inside Search while also extending the model’s use across translation and live voice agents. The update introduces more natural spoken responses in Search Live and reflects Google’s effort to improve natural voice queries, treating voice as a core interface as a way for users to get everything they can get from regular search plus enabling them to ask questions about the physical world around them and receive immediate voice translations between two people speaking different languages.

The new updated voice capabilities, rolling out this week in the  United States, will enable Google’s voice responses to sound more natural and can even be slowed down for instructional content.

According to Google:

“When you go Live with Search, you can have a back-and-forth voice conversation in AI Mode to get real-time help and quickly find relevant sites across the web. And now, thanks to our latest Gemini model for native audio, the responses on Search Live will be more fluid and expressive than ever before.”

Broader Gemini Native Audio Rollout

This Search upgrade is part of a broader update to Gemini 2.5 Flash Native Audio rolling out across Google’s ecosystem, including Gemini Live (in the Gemini App), Google AI Studio, and Vertex AI. The model processes spoken audio in real time and produces fluid spoken responses, reducing barriers to natural conversation, reducing friction in live interactions. Although Google’s announcement didn’t say that the model was a speech-to-speech model (as opposed to speech-to-text then text-to-speech), this update follows Google’s October announcement of “Speech-to-Retrieval (S2R). It’s a neural network-based machine-learning model trained on large datasets of paired audio queries.”

These changes show Google treating native audio as a core capability across consumer-facing products, making it easier for users to ask and receive information about the physical world around them in a natural manner that wasn’t previously possible.

Improvements For Voice-Based Systems

For developers and enterprises building voice-based systems, Google says the updated model improves reliability in several areas. Gemini 2.5 Flash Native Audio more consistently triggers external functions during conversations, follows complex instructions, and maintains context across multiple turns. These improvements make live voice agents more dependable in real-world workflows, where misinterpreted instructions or broken conversational flow reduce usability.

Smooth Conversational Translation

Beyond Search and voice agents, the update introduces native support for “live speech-to-speech translation.” Gemini translates spoken language in real time, either by continuously translating ambient speech into a target language or by handling conversations between speakers of different languages in both directions. The system preserves vocal characteristics such as speech rhythm and emphasis, supporting translation that sounds smoother and conversational.

Google highlights several capabilities supporting this translation feature, including broad language coverage, automatic language detection, multilingual input handling, and noise filtering for everyday environments. These features reduce setup friction and allow translation to occur passively during conversation rather than through manual controls. The result is a translation experience that behaves much like an actual person in the middle translating between two people.

Voice Search Realizing Google’s Aspirations

The update reflects Google’s continued iteration of voice search toward an ideal that was originally inspired by the science fiction voice interactions between humans and computers in the popular Star Trek television and movie series.

Read More:

Google Announces A New Era For Voice Search

You can now have more fluid and expressive conversations when you go Live with Search.

Improved Gemini audio models for powerful voice interactions

Gemini Live

5 ways to get real-time help by going Live with Search

Featured Image by Shutterstock/Jackbin

How People Use Copilot Depends On Device, Microsoft Says via @sejournal, @MattGSouthern

How people use Microsoft Copilot depends on whether they’re at a desk or on their phone.

That is the core theme in the company’s analysis of 37.5 million Copilot conversations sampled between January and September.

The research examines consumer Copilot usage patterns across device types and time of day. The authors say they used machine-based classifiers to categorize conversations by topic and intent without any human review of the messages.

What The Report Says

On mobile, Health and Fitness is the most common topic throughout the day

The authors summarize the split this way:

“On mobile, health is the dominant topic, which is consistent across every hour and every month we observed, with users seeking not just information but also advice.”

Desktop usage follows a different rhythm. Technology leads as the top topic overall, but the researchers report that work-related conversations rise during business hours.

They describe “three distinct modes of interaction: the workday, the constant personal companion, and the introspective night.”

During the workday, the paper says:

  • Between 8 a.m. and 5 p.m., “Work and Career” overtakes “Technology” as the top topic on desktop.
  • Education and science topics rise during business hours compared to nighttime.

Outside business hours, the paper describes a shift toward more personal and reflective topics. For example, it reports that “Religion and Philosophy” rises in rank during late-night hours through dawn.

Programming conversations are more common on weekdays, while gaming rises on weekends. They also note a spike in relationship conversations on Valentine’s Day.

Methodology Caveats

A few limitations are worth keeping in mind.

This is a preprint, so it hasn’t been peer reviewed. It also focuses on consumer Copilot usage and excludes enterprise-authenticated traffic, so it doesn’t describe how Copilot is used inside Microsoft 365 at work.

Finally, the topic and intent labels come from automated classifiers, which means the results reflect how Microsoft’s system groups conversations, not a human-coded review.

Why This Matters

This paper suggests that the use of AI chatbots varies with context. The researchers describe mobile behavior as consistently health-oriented, while desktop behavior is more tied to the workday.

The researchers connect the mobile health pattern to how people use their phones. They write:

“This suggests a device-specific usage pattern where the phone serves as a constant confidant for physical well-being, regardless of the user’s schedule.”

The big takeaway is that “Copilot usage” is not one uniform behavior. Device and time of day appear to shape what people ask for, and how they ask it.

Looking Ahead

Enterprise usage patterns may look different, especially inside Microsoft 365. Any follow-up research that includes workplace contexts, or that validates these patterns outside Microsoft’s own tooling and taxonomy, would help clarify how broadly these findings apply.

Google Releases December 2025 Core Update via @sejournal, @MattGSouthern

Google has released the December 2025 core update, the company confirmed through its Search Status Dashboard.

The rollout began at 9:25 a.m. Pacific Time on December 11, 2025.

This marks Google’s third core update of 2025, following the March and June core updates earlier this year.

What’s New

Google lists the update as an “incident affecting ranking” on its status dashboard.

The company states the rollout “may take up to three weeks to complete.”

Core updates are broad changes to Google’s ranking systems designed to improve search results overall. Unlike specific updates targeting spam or particular ranking factors, core updates affect how Google’s systems assess content across the web.

2025 Core Update Timeline

The December update follows two previous core updates this year.

The March 2025 core update rolled out from March 13-27, taking 14 days to complete. Data from SEO tracking providers suggested volatility similar to the December 2024 core update.

The June 2025 core update ran from June 30 to July 17, lasting about 16 days. SEO data providers indicated it was one of the larger core updates in recent memory. Some sites previously hit by the September 2023 Helpful Content Update saw partial recoveries during this rollout.

Documentation Update On Continuous Changes

Two days before this core update, Google updated its core updates documentation with new language about ongoing algorithm changes.

The updated documentation now states:

“However, you don’t necessarily have to wait for a major core update to see the effect of your improvements. We’re continually making updates to our search algorithms, including smaller core updates. These updates are not announced because they aren’t widely noticeable, but they are another way that your content can see a rise in position (if you’ve made improvements).”

Google explained that the addition was meant to clarify that content improvements can lead to ranking changes without waiting for the next announced update.

Why This Matters

If you notice ranking fluctuations over the coming weeks, this update is likely a major factor.

Core updates can shift rankings for pages that weren’t doing anything wrong. Google has consistently stated that pages losing visibility after a core update don’t necessarily have problems to fix. The systems are reassessing content relative to what else is available.

The documentation update is a reminder that rankings can change between major updates as Google rolls out smaller core changes behind the scenes.

Looking Ahead

Google will update the Search Status Dashboard when the rollout is complete.

Monitor your rankings and traffic over the next three weeks. If you see changes, document when they occurred relative to the rollout timeline.

Based on 2025’s previous updates, completion typically takes two to three weeks. Google will confirm completion through the dashboard and its Search Central social accounts.

YouTube Adds Comments To Shorts Ads, Expands To Mobile Web via @sejournal, @MattGSouthern

YouTube adds comment sections to eligible Shorts ads, lets creators link to brand websites, and expands Shorts ads to mobile web browsers.

  • Eligible Shorts ads can now display comment sections, matching the experience of organic Shorts content.
  • Shorts creators can link directly to brand websites when producing branded content.
  • Shorts ads now serve on mobile web browsers, extending beyond the mobile app.
Google Expands Preferred Sources & Publisher AI Partnerships via @sejournal, @MattGSouthern

Google is expanding its Preferred Sources feature to English-language users worldwide and launching a pilot program to test AI-powered features with major news publishers.

The announcement includes updates to how links appear in AI Mode and a new feature that will highlight content from users’ news subscriptions.

Preferred Sources Goes Global

Preferred Sources in Search lets users customize Top Stories to see more from their favorite outlets. Google is now rolling it out globally for English-language users, with all supported languages following early next year.

Google shared usage data from the feature’s initial rollout. Nearly 90,000 unique sources have been selected by users, ranging from local blogs to global news outlets. Users who pick a preferred source click to that site twice as often on average.

Subscription Highlighting

A new feature will highlight links from users’ paid news subscriptions in search results. Google will also prioritize links from subscribed publications and show them in a dedicated carousel.

The feature launches first in the Gemini app in the coming weeks. AI Overviews and AI Mode will follow, though Google didn’t provide a timeline.

AI Mode Link Updates

Google is increasing the number of inline links in AI Mode and updating their design. The company is also adding contextual introductions to embedded links. These are short statements explaining why a particular link might be useful.

Web Guide, which organizes links into topic groups using AI, is now twice as fast and appearing on more searches for users opted into the experiment.

Publisher AI Pilot Program

Google announced a commercial partnership pilot with publishers including Der Spiegel, El País, Folha de S. Paulo, Infobae, Kompas, The Guardian, The Times of India, The Washington Examiner, and The Washington Post.

The pilot will test AI-powered features in Google News. These include article overviews on participating publications’ Google News pages and audio briefings for those who prefer listening. Google says these features will include attribution and link to articles.

Separate partnerships with Estadão, Antara, Yonhap, and The Associated Press will provide real-time information for the Gemini app.

Google says it has partnered with over 3,000 publications, platforms, and content providers in more than 50 countries in the last few years.

Why This Matters

If you’ve been watching how Google handles publisher relationships in the AI era, this announcement outlines their current approach. The Preferred Sources data suggests users who customize their sources engage more with those sites.

The subscription highlighting feature could affect how your subscribed audiences find your content across Google’s surfaces.

Looking Ahead

Preferred Sources is available now for English-language users globally. Full language support arrives early 2026.

The subscription highlighting feature starts in the Gemini app in the coming weeks. The publisher AI pilot has begun with participating publications in Google News. Google didn’t provide timelines for when AI Mode and AI Overviews will get subscription highlighting.

Google Hit By EU Probe Into Unfair Use Of Online Content via @sejournal, @martinibuster

The European Commission has launched an antitrust inquiry into Google to determine whether the company has violated EU competition rules, partly focusing on whether Google has used creator and publisher content in ways that leave publishers unable to refuse such use without risking their search traffic. It is also looking into whether Google is granting itself privileged access to YouTube content for AI in a way that leaves competitors at a disadvantage.

How Google’s Terms May Pressure Publishers and Creators

The Commission is focusing on publisher content is used by AI Overviews and AI Mode to generate answers but without a way to compensate the publishers or for them to opt out of having their content used to generate summaries.

They write:

“The Commission will investigate to what extent the generation of AI Overviews and AI Mode by Google is based on web publishers’ content without appropriate compensation for that, and without the possibility for publishers to refuse without losing access to Google Search. Indeed, many publishers depend on Google Search for user traffic, and they do not want to risk losing access to it.”

This raises concerns that Google may be using publisher content in its AI products without offering a workable opt-out, leaving publishers who rely on Search traffic with little choice but to accept this use.

Use of YouTube Content to Train Google’s AI Models

The Commission is also examining Google’s use of YouTube videos and other creator content for training its generative AI models. According to the announcement, creators “have an obligation to grant Google permission to use their data for different purposes, including for training generative AI models,” and cannot upload content while withholding that permission. Google provides no payment for this use while blocking rival AI developers from training on YouTube content under YouTube’s policies.

This mix of mandatory access for Google, limits on competitors, and no payment for creators underpins the Commission’s concern that Google may be giving itself preferred access to YouTube content in a way that may harm the wider AI market.

The Commission has notified Google that it has opened an investigation into whether they have breached EU competition rules prohibiting the abuse of a dominant position.

Featured Image by Shutterstock/Mo Arbid

Google Confirms Smaller Core Updates Happen Continuously via @sejournal, @MattGSouthern

Google updated its core updates documentation to say smaller core updates happen on an ongoing basis, so sites can improve without waiting for named updates.

  • Google explicitly confirms it makes “smaller core updates” beyond the named updates announced several times per year.
  • Sites that improve their content can see ranking gains without waiting for the next major core update to roll out.
  • The documentation change addresses whether recovery between named updates is possible.
YouTube AI Enforcement Questioned As Channels Get Restored via @sejournal, @MattGSouthern

YouTube creators are raising concerns about the platform’s AI-driven moderation system. Multiple accounts describe sudden channel terminations for “spam, deceptive practices and scams,” followed by rapid appeal rejections with templated responses.

In some cases, channels have been restored only after the creator generated attention on X or Reddit. YouTube’s message to creators states the company has “not identified any widespread issues” with channel terminations and says only “a small percentage” of enforcement actions are reversed.

There’s a gap between YouTube’s position and creator experiences that’s driving a debate.

What Creators Are Reporting

The pattern appearing across X and Reddit threads follows a similar sequence.

Channels receive termination notices citing “spam, deceptive practices and scams.” Appeals get rejected within hours, sometimes minutes, with generic language. When channels are restored, creators say they receive no explanation of what triggered the ban or how to prevent future issues.

One documented case comes from YouTube creator “Chase Car,” who runs an EV news channel. In a detailed post on r/YouTubeCreators, they describe a sequence where their channel was demonetized by an automated system, cleared by a human reviewer, then terminated months later for spam.

The creator says they escalated the case to an EU-certified dispute body under the Digital Services Act. According to their account, the decision found the termination “was not rightful.” As of their most recent update, YouTube had not acted on the ruling.

Channels Restored After Public Attention

A subset of terminated channels have been reinstated after their cases gained visibility on social media.

Film analysis channel Final Verdict shared a thread documenting a sudden spam-related termination and later reinstatement after posts on X gained traction.

True crime channel The Dark Archive had their channel removed and later restored after tagging TeamYouTube publicly.

Streamer ProkoTV said their channel was restricted from live streaming after a spam warning. TeamYouTube later acknowledged an error and restored access.

These reversals confirm that some enforcement actions are incorrect by YouTube’s own standards. They also suggest that escalation on X can function as a parallel appeal route.

YouTube Acknowledges Some Errors

In a few cases, YouTube or its representatives have publicly admitted mistakes.

Dexerto reported on a creator whose 100,000-plus subscriber channel was banned over a comment they wrote on a different account at age 13. YouTube eventually apologized, telling the creator the ban “was a mistake on our end.”

Tech YouTuber Enderman, with 350,000 subscribers, said an automated system shut down their channel after linking it to an unrelated banned account. Dexerto highlighted the case after it spread on X.

YouTube’s Official Position

YouTube frames its enforcement differently than creators describe.

The company’s spam, deceptive practices, and scams policy explains why it takes action on fraud, impersonation, fake engagement, and misleading metadata. The policy notes that YouTube may act at the channel level if an account exists “primarily” to violate rules.

In a FAQ post, YouTube says the “vast majority” of terminations are upheld on appeal. The company says it’s “confident” in its processes while acknowledging “a handful” of incorrect terminations that were later reversed.

YouTube also offers a “Second Chances” pilot program that allows some creators to start new channels if they meet specific criteria and were terminated more than a year ago. The program doesn’t restore lost videos or subscribers.

YouTube’s CEO recently indicated the company plans to expand AI moderation tools. In an interview with Time, he said YouTube will proceed with expanded AI enforcement despite creator concerns.

Why This Matters

If you rely on YouTube as a core channel, these accounts raise practical concerns. A channel termination removes your entire presence, including subscribers and revenue potential. When appeals feel automated, you have limited visibility into what triggered the enforcement.

The Chase Car timeline shows an AI system can overturn a positive human verdict months later. Creators without large followings may have fewer options for escalation if formal appeals fail.

Looking Ahead

The EU’s Digital Services Act gives European users access to certified dispute bodies for moderation decisions. The Chase Car case could test how platforms respond to unfavorable rulings under that system.

YouTube says its appeals process is the correct channel for enforcement disputes. The company has not announced changes to its moderation approach in response to creator complaints.

Monitor YouTube’s official help community for any updates to appeal procedures or policy clarifications.


Featured Image: T. Schneider/Shutterstock

Google Disputes Report Claiming Ads Are Coming To Gemini In 2026 via @sejournal, @MattGSouthern

Google is publicly pushing back on an Adweek report that claimed the company told advertising clients it plans to bring ads to its Gemini AI chatbot next year.

Dan Taylor, Google’s Vice President of Global Ads, responded directly on X shortly after the story published, calling the report inaccurate and denying any plans to monetize the Gemini app.

The Original Report

Adweek’s Trishla Ostwal reported that Google had informed advertising clients about plans to introduce ads to Gemini. According to the exclusive story, Google representatives held calls with at least two advertising clients indicating that ad placements in Gemini were targeted for a 2026 rollout.

The agency buyers who spoke to Adweek remained anonymous. They said details on ad formats, pricing, and testing were unclear, and that Google had not shared prototypes or technical specifications about how ads would appear in the chatbot.

Notably, the report said this plan would be separate from advertisements in AI Mode, Google’s AI-powered search experience.

Google’s Response

Taylor disputed the claims publicly on X, writing: “This story is based on uninformed, anonymous sources who are making inaccurate claims. There are no ads in the Gemini app and there are no current plans to change that.”

Google’s official AdsLiaison account amplified the denial, reiterating that there are no ads in the Gemini app and no current plans to add them, and pointing out that ads currently appear in AI Overviews in English in the US, with expansion to more English-speaking countries, and are being tested in AI Mode.

Logan Kilpatrick, who works on Google’s Gemini team, responded to Taylor’s post with “thanks for clarifying!!”

Where Google Is Monetizing AI

While the Gemini app itself remains ad-free according to Google, the company is actively monetizing other AI-powered search experiences.

Google began showing ads in AI Overviews earlier this year and has been expanding that program to additional English-speaking countries. The company also continues testing advertisements within AI Mode.

Why This Matters

The question of how AI chatbots will be monetized has become increasingly relevant as these products gain mainstream adoption. Google, OpenAI, and other AI companies face pressure to generate revenue from expensive-to-run conversational AI products.

Just last week, code discovered in ChatGPT’s Android app suggested OpenAI may be building an advertising framework, though the company has not confirmed any plans to introduce ads.

For now, Google maintains that Gemini users won’t see ads in the chatbot app. Whether that position changes as the AI landscape evolves remains to be seen.