In an interview with Kenichi Suzuki from Faber Company Inc., Google Developer Advocate Martin Splitt recently shared key information about JavaScript rendering, server-side vs. client-side rendering, and structured data.
The talk cleared up common SEO confusion and offered practical tips for developers and marketers working with Google’s changing search systems.
Google’s AI Crawler & JavaScript Rendering
When asked how AI systems handle JavaScript content, Splitt revealed that Google’s AI crawler (used by Gemini) processes JavaScript well through a shared service.
Splitt explained:
“We don’t share what Googlebot sees for web search, but Google’s AI crawler that Gemini uses also renders. It uses WRS [Web Rendering Service], but it’s basically like we have a service Googlebot uses, and Gemini uses the service as well.”
This gives Google’s AI tools an edge over competitors that have trouble with JavaScript.
While one study mentioned in the interview claimed rendering sometimes takes weeks, Splitt explained that it usually happens much faster.
“The 99th percentile is within minutes,” Splitt noted, suggesting that long delays are rare and might be due to measurement errors.
Server-Side vs. Client-Side Rendering: Which is Better?
Part of the discussion covered the debate between server-side rendering (SSR) and client-side rendering (CSR).
Instead of saying one is always better, Splitt stressed that the right choice depends on what your website does.
Splitt stated:
“If you have a website that is a classical website that is basically just presenting information to the user, then requiring JavaScript is a drawback. It can break. It can cause problems. It will make things slower. It will need more battery on your phone.”
Splitt suggests SSR or even pre-rendering static HTML for websites focused on content. But CSR works better for interactive tools like CAD programs or video editors.
Splitt clarified:
“It’s not one or the other. It is two tools. Do you need a hammer or do you need a screwdriver? That depends on what you’re trying to do.”
The talk then moved to structured data, which is becoming more important as AI systems grow in search.
When asked if structured data helps Google’s AI understand content better, like Microsoft claims about Bing, Splitt confirmed it helps.
He stated:
“Structured data gives us more information and gives us more confidence in information. So it makes sense to have structured data.”
However, Splitt clarified that while structured data adds context, it “does not push rankings” directly. This is an important difference for SEO professionals who might think it directly boosts search positions.
What This Means
Here are the key things we learned from this interview:
Google’s rendering usually happens within minutes, so the old fear of JavaScript-heavy sites being at a disadvantage is less of an issue now.
Non-Google AI tools may still have trouble with JavaScript, making SSR possibly more critical for visibility across all AI systems.
Use SSR for content sites and CSR for interactive tools. Don’t use one solution for everything.
Though not a ranking factor, structured data helps Google understand your content better. This matters more as AI becomes a bigger part of search.
In his final advice to SEO professionals, Splitt highlighted basic principles over technical tricks:
“Think about your users. Figure out what is your business goal, how to make users happy, and then just create great content.”
As AI changes search technology, understanding these technical details becomes more important for marketers who want to optimize content for people and search algorithms.
It took me more than a decade to discover all these lessons. By reading this article, you can apply these insights to save yourself and your SEO clients time, money, and frustration – in less than an hour.
Lesson #1: Technical SEO Is Your Foundation For SEO Success
Lesson: You should always start any SEO work with technical fundamentals; crawlability and indexability determine whether search engines can even see your site.
Technical SEO ensures search engines can crawl, index, and fully understand your content. If search engines can’t properly access your site, no amount of quality content or backlinks will help.
After auditing over 500 websites, I believe technical SEO is the most critical aspect of SEO, which comes down to two fundamental concepts:
Crawlability: Can search engines easily find and navigate your website’s pages?
Indexability: Once crawled, can your pages appear in search results?
If your pages fail these two tests, they won’t even enter the SEO game — and your SEO efforts won’t matter.
Lesson: You should be cautious when relying heavily on JavaScript. It can easily prevent Google from seeing and indexing critical content.
JavaScript adds great interactivity, but search engines (even as smart as Google) often struggle to process it reliably.
Google handles JavaScript in three steps (crawling, rendering, and indexing) using an evergreen Chromium browser. However, rendering delays (from minutes to weeks) and limited resources can prevent important content from getting indexed.
I’ve audited many sites whose SEO was failing because key JavaScript-loaded content wasn’t visible to Google.
Typically, important content was missing from the initial HTML, it didn’t load properly during rendering, or there were significant differences between the raw HTML and rendered HTML when it came to content or meta elements.
You should always test if Google can see your JavaScript-based content:
Use the Live URL Test in Google Search Console and verify rendered HTML.
Google Search Console LIVE Test allows you to see the rendered HTML. (Screenshot from Google Search Console, April 2025)
Or, search Google for a unique sentence from your JavaScript content (in quotes). If your content isn’t showing up, Google probably can’t index it.*
The site: search in Google allows you to quickly check whether a given piece of text on a given page is indexed by Google. (Screenshot from Google Search, April 2025)
*This will only work for URLs that are already in Google’s index.
Here are a few best practices regarding JavaScript SEO:
Critical content in HTML: You should include titles, descriptions, and important content directly in the initial HTML so search engines can index it immediately. You should remember that Google doesn’t scroll or click.
Server-Side Rendering (SSR): You should consider implementing SSR to serve fully rendered HTML. It’s more reliable and less resource-intensive for search engines.
Proper robots.txt setup: Websites should block essential JavaScript files needed for rendering, as this prevents indexing.
Use crawlable URLs: You should ensure each page has a unique, crawlable URL. You should also avoid URL fragments (#section) for important content; they often don’t get indexed.
Lesson #3: Crawl Budget Matters, But Only If Your Website Is Huge
Lesson: You should only worry about the crawl budget if your website has hundreds of thousands or millions of pages.
Crawl budget refers to how many pages a search engine like Google crawls on your site within a certain timeframe. It’s determined by two main factors:
Crawl capacity limit: This prevents Googlebot from overwhelming your server with too many simultaneous requests.
Crawl demand: This is based on your site’s popularity and how often content changes.
No matter what you hear or read on the internet, most websites don’t need to stress about crawl budget at all. Google typically handles crawling efficiently for smaller websites.
But for huge websites – especially those with millions of URLs or daily-changing content – crawl budget becomes critical (as Google confirms in its crawl budget documentation).
Google, in its documentation, clearly defines what types of websites should be concerned about crawl budget. (Screenshot from Search Central, April 2025)
In this case, you need to ensure that Google prioritizes and crawls important pages frequently without wasting resources on pages that should never be crawled or indexed.
You can check your crawl budget health using Google Search Console’s Indexing report. Pay attention to:
Crawled – Currently Not Indexed: This usually indicates indexing problems, not crawl budget.
Discovered – Currently Not Indexed: This typically signals crawl budget issues.
You should also regularly review Google Search Console’s Crawl Stats report to see how many pages Google crawls per day. Comparing crawled pages with total pages on your site helps you spot inefficiencies.
While those quick checks in GSC naturally won’t replace log file analysis, they will give quick insights into possible crawl budget issues and may suggest that a detailed log file analysis may be necessary.
Lesson #4: Log File Analysis Lets You See The Entire Picture
Lesson: Log file analysis is a must for many websites. It reveals details you can’t see otherwise and helps diagnose problems with crawlability and indexability that affect your site’s ability to rank.
Log files track every visit from search engine bots, like Googlebot or Bingbot. They show which pages are crawled, how often, and what the bots do. This data lets you spot issues and decide how to fix them.
For example, on an ecommerce site, you might find Googlebot crawling product pages, adding items to the cart, and removing them, wasting your crawl budget on useless actions.
With this insight, you can block those cart-related URLs with parameters to save resources so that Googlebot can crawl and index valuable, indexable canonical URLs.
Here is how you can make use of log file analysis:
Start by accessing your server access logs, which record bot activity.
Look at what pages bots hit most, how frequently they visit, and if they’re stuck on low-value URLs.
You don’t need to analyze logs manually. Tools like Screaming Frog Log File Analyzer make it easy to identify patterns quickly.
If you notice issues, like bots repeatedly crawling URLs with parameters, you can easily update your robots.txt file to block those unnecessary crawls
Getting log files isn’t always easy, especially for big enterprise sites where server access might be restricted.
If that’s the case, you can use the aforementioned Google Search Console’s Crawl Stats, which provides valuable insights into Googlebot’s crawling activity, including pages crawled, crawl frequency, and response times.
The Google Search Console Crawl Stats report provides a sample of data about Google’s crawling activity. (Screenshot from Google Search Console, April 2025)
While log files offer the most detailed view of search engine interactions, even a quick check in Crawl Stats helps you spot issues you might otherwise miss.
Lesson #5: Core Web Vitals Are Overrated. Stop Obsessing Over Them
Lesson: You should focus less on Core Web Vitals. They rarely make or break SEO results.
Core Web Vitals measure loading speed, interactivity, and visual stability, but they do not influence SEO as significantly as many assume.
After auditing over 500 websites, I’ve rarely seen Core Web Vitals alone significantly improve rankings.
Most sites only see measurable improvement if their loading times are extremely poor – taking more than 30 seconds – or have critical issues flagged in Google Search Console (where everything is marked in red).
The Core Web Vitals report in Google Search Console provides real-world user data. (Screenshot from Google Search Console, April 2025)
I’ve watched clients spend thousands, even tens of thousands of dollars, chasing perfect Core Web Vitals scores while overlooking fundamental SEO basics, such as content quality or keyword strategy.
Redirecting those resources toward content and foundational SEO improvements usually yields way better results.
When evaluating Core Web Vitals, you should focus exclusively on real-world data from Google Search Console (as opposed to lab data in Google PageSpeed Insights) and consider users’ geographic locations and typical internet speeds.
If your users live in urban areas with reliable high-speed internet, Core Web Vitals won’t affect them much. But if they’re rural users on slower connections or older devices, site speed and visual stability become critical.
The bottom line here is that you should always base your decision to optimize Core Web Vitals on your specific audience’s needs and real user data – not just industry trends.
Lesson #6: Use Schema (Structured Data) To Help Google Understand & Trust You
Lesson: You should use structured data (Schema) to tell Google who you are, what you do, and why your website deserves trust and visibility.
Schema Markup (or structured data) explicitly defines your content’s meaning, which helps Google easily understand the main topic and context of your pages.
Certain schema types, like rich results markup, allow your listings to display extra details, such as star ratings, event information, or product prices. These “rich snippets” can grab attention in search results and increase click-through rates.
You can think of schema as informative labels for Google. You can label almost anything – products, articles, reviews, events – to clearly explain relationships and context. This clarity helps search engines understand why your content is relevant for a given query.
You should always choose the correct schema type (like “Article” for blog posts or “Product” for e-commerce pages), implement it properly with JSON-LD, and carefully test it using Google’s Rich Results Test or Structured Data Testing Tool.
In its documentation, Google shows examples of structured data markup supported by Google Search. (Screenshot from Google Search Console, April 2025)
Schema lets you optimize SEO behind the scenes without affecting what your audience sees.
While SEO clients often hesitate about changing visible content, they usually feel comfortable adding structured data because it’s invisible to website visitors.
Lesson #7: Keyword Research And Mapping Are Everything
Lesson: Technical SEO gets you into the game by controlling what search engines can crawl and index. But, the next step – keyword research and mapping – tells them what your site is about and how to rank it.
Too often, websites chase the latest SEO tricks or target broad, competitive keywords without any strategic planning. They skip proper keyword research and rarely invest in keyword mapping, both essential steps to long-term SEO success:
Keyword research identifies the exact words and phrases your audience actually uses to search.
Keyword mapping assigns these researched terms to specific pages and gives each page a clear, focused purpose.
Every website should have a spreadsheet listing all its indexable canonical URLs.
Next to each URL, there should be the main keyword that the page should target, plus a few related synonyms or variations.
Having the keyword mapping document is a vital element of any SEO strategy. (Image from author, April 2025)
Without this structure, you’ll be guessing and hoping your pages rank for terms that may not even match your content.
A clear keyword map ensures every page has a defined role, which makes your entire SEO strategy more effective.
This isn’t busywork; it’s the foundation of a solid SEO strategy.
Lesson #8: On-Page SEO Accounts For 80% Of Success
Lesson: From my experience auditing hundreds of websites, on-page SEO drives about 80% of SEO results. Yet, only about 1 in 20 or 30 sites I review have done it well. Most get it wrong from the start.
Many websites rush straight into link building, generating hundreds or even thousands of low-quality backlinks with exact-match anchor texts, before laying any SEO groundwork.
They skip essential keyword research, overlook keyword mapping, and fail to optimize their key pages first.
I’ve seen this over and over: chasing advanced or shiny tactics while ignoring the basics that actually work.
When your technical SEO foundation is strong, focusing on on-page SEO can often deliver significant results.
There are thousands of articles about basic on-page SEO: optimizing titles, headers, and content around targeted keywords.
Yet, almost nobody implements all of these basics correctly. Instead of chasing trendy or complex tactics, you should focus first on the essentials:
Do proper keyword research to identify terms your audience actually searches.
Map these keywords clearly to specific pages.
Optimize each page’s title tags, meta descriptions, headers, images, internal links, and content accordingly.
These straightforward steps are often enough to achieve SEO success, yet many overlook them while searching for complicated shortcuts.
Lesson #9: Internal Linking Is An Underused But Powerful SEO Opportunity
Lesson: Internal links hold more power than overhyped external backlinks and can significantly clarify your site’s structure for Google.
Internal links are way more powerful than most website owners realize.
Everyone talks about backlinks from external sites, but internal linking – when done correctly – can actually make a huge impact.
Unless your website is brand new, improving your internal linking can give your SEO a serious lift by helping Google clearly understand the topic and context of your site and its specific pages.
Still, many websites don’t use internal links effectively. They rely heavily on generic anchor texts like “Read more” or “Learn more,” which tell search engines absolutely nothing about the linked page’s content.
Image from author, April 2025
Website owners often approach me convinced they need a deep technical audit.
Yet, when I take a closer look, their real issue frequently turns out to be poor internal linking or unclear website structure, both making it harder for Google to understand the site’s content and value.
Internal linking can also give a boost to underperforming pages.
For example, if you have a page with strong external backlinks, linking internally from that high-authority page to weaker ones can pass authority and help those pages rank better.
Investing a little extra time in improving your internal links is always worth it. They’re one of the easiest yet most powerful SEO tools you have.
Lesson #10: Backlinks Are Just One SEO Lever, Not The Only One
Lesson: You should never blindly chase backlinks to fix your SEO. Build them strategically only after mastering the basics.
SEO audits often show websites placing too much emphasis on backlinks while neglecting many other critical SEO opportunities.
Blindly building backlinks without first covering SEO fundamentals – like removing technical SEO blockages, doing thorough keyword research, and mapping clear keywords to every page – is a common and costly mistake.
Even after getting those basics right, link building should never be random or reactive.
Too often, I see sites start building backlinks simply because their SEO isn’t progressing, hoping more links will magically help. This rarely works.
Instead, you should always approach link building strategically, by first carefully analyzing your direct SERP competitors to determine if backlinks are genuinely your missing element:
Look closely at the pages outranking you.
Identify whether their advantage truly comes from backlinks or better on-page optimization, content quality, or internal linking.
The decision on whether or not to build backlinks should be based on whether direct competitors have more and better backlinks. (Image from author, April 2025)
Only after ensuring your on-page SEO and internal links are strong and confirming that backlinks are indeed the differentiating factor, should you invest in targeted link building.
Typically, you don’t need hundreds of low-quality backlinks. Often, just a few strategic editorial links or well-crafted SEO press releases can close the gap and improve your rankings.
Lesson #11: SEO Tools Alone Can’t Replace Manual SEO Checks
Lesson: You should never trust SEO tools blindly. Always cross-check their findings manually using your own judgment and common sense.
SEO tools make our work faster, easier, and more efficient, but they still can’t fully replicate human analysis or insight.
Tools lack the ability to understand context and strategy in the way that SEO professionals do. They usually can’t “connect the dots” or assess the real significance of certain findings.
This is exactly why every recommendation provided by a tool needs manual verification. You should always evaluate the severity and real-world impact of the issue yourself.
Often, website owners come to me alarmed by “fatal” errors flagged by their SEO tools.
Yet, when I manually inspect these issues, most turn out to be minor or irrelevant.
Meanwhile, fundamental aspects of SEO, such as strategic keyword targeting or on-page optimization, are completely missing since no tool can fully capture these nuances.
Screaming Frog SEO Spider says there are rich result validation errors, but when I check that manually, there are no errors. (Screenshot from Screaming Frog, April 2025)
SEO tools are still incredibly useful because they handle large-scale checks that humans can’t easily perform, like analyzing millions of URLs at once.
However, you should always interpret their findings carefully and manually verify the importance and actual impact before taking any action.
Final Thoughts
After auditing hundreds of websites, the biggest pattern I notice isn’t complex technical SEO issues, though they do matter.
Instead, the most frequent and significant problem is simply a lack of a clear, prioritized SEO strategy.
Too often, SEO is done without a solid foundation or clear direction, which makes all other efforts less effective.
Another common issue is undiagnosed technical problems lingering from old site migrations or updates. These hidden problems can quietly hurt rankings for years if left unresolved.
The lessons above cover the majority of challenges I encounter daily, but remember: Each website is unique. There’s no one-size-fits-all checklist.
Every audit must be personalized and consider the site’s specific context, audience, goals, and limitations.
SEO tools and AI are increasingly helpful, but they’re still just tools. Ultimately, your own human judgment, experience, and common sense remain the most critical factors in effective SEO.
The humble robots.txt file often sits quietly in the background of a WordPress site, but the default is somewhat basic out of the box and, of course, doesn’t contribute towards any customized directives you may want to adopt.
No more intro needed – let’s dive right into what else you can include to improve it.
(A small note to add: This post is only useful for WordPress installations on the root directory of a domain or subdomain only, e.g., domain.com or example.domain.com. )
Where Exactly Is The WordPress Robots.txt File?
By default, WordPress generates a virtual robots.txt file. You can see it by visiting /robots.txt of your install, for example:
https://yoursite.com/robots.txt
This default file exists only in memory and isn’t represented by a file on your server.
If you want to use a custom robots.txt file, all you have to do is upload one to the root folder of the install.
You can do this either by using an FTP application or a plugin, such as Yoast SEO (SEO → Tools → File Editor), that includes a robots.txt editor that you can access within the WordPress admin area.
The Default WordPress Robots.txt (And Why It’s Not Enough)
If you don’t manually create a robots.txt file, WordPress’ default output looks like this:
There are now dated suggestions to disallow some core WordPress directories like /wp-includes/, /wp-content/plugins/, or even /wp-content/uploads/. Don’t!
Here’s why you shouldn’t block them:
Google is smart enough to ignore irrelevant files. Blocking CSS and JavaScript can hurt renderability and cause indexing issues.
You may unintentionally block valuable images/videos/other media, especially those loaded from /wp-content/uploads/, which contains all uploaded media that you definitely want crawled.
Instead, let crawlers fetch the CSS, JavaScript, and images they need for proper rendering.
Managing Staging Sites
It’s advisable to ensure that staging sites are not crawled for both SEO and general security purposes.
You should still use the noindex meta tag, but to ensure another layer is covered, it’s still advisable to do both.
If you navigate to Settings > Reading, you can tick the option “Discourage search engines from indexing this site,” which does the following in the robots.txt file (or you can add this in yourself).
User-agent: *
Disallow: /
Google may still index pages if it discovers links elsewhere (usually caused by calls to staging from production when migration isn’t perfect).
Important: When you move to production, ensure you double-check this setting again to ensure that you revert any disallowing or noindexing.
Clean Up Some Non-Essential Core WordPress Paths
Not everything should be blocked, but many default paths add no SEO value, such as the below:
Sometimes, you’ll want to stop search engines from crawling URLs with known low-value query parameters, like tracking parameters, comment responses, or print versions.
You can use Google Search Console’s URL Parameters tool to monitor parameter-driven indexing patterns and decide if additional disallows are worthy of adding.
Disallowing Low-Value Taxonomies And SERPs
If your WordPress site includes tag archives or internal search results pages that offer no added value, you can block them too:
If you use tag taxonomy pages as part of content you want indexed and crawled, then ignore this, but generally, they don’t add any benefits.
Also, make sure your internal linking structure supports your decision and minimizes any internal linking to areas you have no intention of indexing or crawling.
Monitor On Crawl Stats
Once your robots.txt is in place, monitor crawl stats via Google Search Console:
Look at Crawl Stats under Settings to see if bots are wasting resources.
Use the URL Inspection Tool to confirm whether a blocked URL is indexed or not.
Check Sitemaps and make sure they only reference pages you actually want crawled and indexed.
In addition, some server management tools, such as Plesk, cPanel, and Cloudflare, can provide extremely detailed crawl statistics beyond Google.
Lastly, use Screaming Frog’s configuration override to simulate changes and revisit Yoast SEO’s crawl optimization features, some of which solve the above.
Final Thoughts
While WordPress is a great CMS, it isn’t set up with the most ideal default robots.txt or set up with crawl optimization in mind.
Just a few lines of code and less than 30 minutes of your time can save you thousands of unnecessary crawl requests to your site that aren’t worthy of being identified at all, as well as securing a potential scaling issue in the future.
In this article, we’ll show you how to rank in AIO SERPs by creating embeddings for your content and how to realign your content for maximum visibility by using AIO tools built by search engineers.
The 3 Key Features Of AI Overviews That Can Make Or Break Your Rankings
Let’s start with the basic building blocks of a Google AI Overviews (AIO) response:
What Are Embeddings?
Embeddings are high-dimensional numerical representations of text. They allow AI systems to understand the meaning of words, phrases, or even entire pages, beyond just the words themselves.
Rather than matching exact terms, embeddings turn language into vectors, or arrays of numbers, that capture the semantic relationships between concepts.
For example, “car,” “vehicle,” and “automobile” are different words, but their embeddings will be close in vector space because they mean similar things.
Large language models (LLMs) like ChatGPT or Google Gemini use embeddings to “understand” language; they don’t just see words, they see patterns of meaning.
Image created by MarketBrew.ai, April 2025
Why Do Embeddings Matter For SEO?
Understanding how Large Language Models (LLMs) interpret content is key to winning in AI-driven search results, especially with Google’s AI Overviews.
Search engines have shifted from simple keyword matching to deeper semantic understanding. Now, they rank content based on contextual relevance, topic clusters, and semantic similarity to user intent, not just isolated words.
Image created by MarketBrew.ai, April 2025
Embeddings power this evolution.
They enable search engines to group, compare, and rank content with a level of precision that traditional methods (like TF-IDF,keyword density, or Entity SEO) can’t match.
By learning how embeddings work, SEOs gain tools to align their content with how search engines actually think, opening the door to better rankings in semantic search.
API Key: Once logged in, you’ll need to generate an API key to access OpenAI’s services. You can find this in your account settings under the API section.
Step 2: Install The OpenAI Python Client To Simplify This Step For SEO Pros
OpenAI provides a Python client that simplifies the process of interacting with their API. To install it, run the following command in your terminal or command prompt:
pip install openai
Step 3: Authenticate With Your API Key
Before making requests, you need to authenticate using your API key. Here’s how you can set it up in your Python script:
At the time of this article’s creation, OpenAI’s text-embedding-3-small is considered one of the most advanced embedding models. It is highly efficient for a wide range of text processing tasks.
Step 5: Create Embeddings For Your Content
To generate embeddings for text:
response = openai.Embedding.create(
model="text-embedding-3-small",
input="This is an example sentence."
)
embeddings = response['data'][0]['embedding']
print(embeddings)
The result is a list of numbers representing the semantic meaning of your input in high-dimensional space.
Step 6: Storing Embeddings
Store embeddings in a database for future use; tools like Pinecone or PostgreSQL with pgvector are great options.
Step 7: Handling Large Text Inputs
For large content, break it down into paragraphs or sections and generate embeddings for each chunk.
Use similarly sized chunks for better cosine similarity calculations. To represent an entire document, you can average the embeddings for each chunk.
💡Pro Tip: Use Market Brew’s free AI Overviews Visualizer. The search engineer team at Market Brew has created this visualizer to help you understand exactly how embeddings, the fourth generation of text classifiers, are used by search engines.
Semantics: Comparing Embeddings With Cosine Similarity
This is essential for comparing the semantic similarity between two pieces of text.
Image created by MarketBrew.ai, April 2025
Typical search engine comparisons include:
Keywords with paragraphs,
Groups of paragraphs with other paragraphs, and
Groups of keywords with groups of paragraphs.
Next, search engines cluster these embeddings.
How Search Engines Cluster Embeddings
Search engines can organize content based on clusters of embeddings.
In the video below, we are going to illustrate why and how you can use embedding clusters, using Market Brew’s free AI Overviews Visualizer, to fix content alignment issues that may be preventing you from appearing in Google’s AI Overviews or even their regular search results!
Embedding clusters, or “semantic clouds”, form one of the most powerful ranking tools for search engineers today.
Semantic clouds are topic clusters in thousands of dimensions. The illustration above shows a 3D representation to simplify understanding.
Topic clusters are to entities as semantic clouds are to embeddings. Think of a semantic cloud as a topic cluster on steroids.
Search engineers use this like they do topic clusters.
When your content falls outside the top semantic cloud – what the AI deems most relevant – it is ignored, demoted, or excluded from AI Overviews (and even regular search results) entirely.
No matter how well-written or optimized your page might be in the traditional sense, it won’t surface if it doesn’t align with the right semantic cluster that the finely tuned AI system is seeking.
By using the AI Overviews Visualizer, you can finally see whether your content aligns with the dominant semantic cloud for a given query. If it doesn’t, the tool provides a realignment strategy to help you bridge that gap.
In a world where AI decides what gets shown, this level of visibility isn’t just helpful. It’s essential.
Free AI Overviews Visualizer: How To Fix Content Alignment
Step 1: Use The Visualizer
Input your URL into this AI Overviews Visualizer tool to see how search engines view your content using embeddings. The Cluster Analysis tab will display embedding clusters for your page and indicate whether your content aligns with the correct cluster.
Screenshot from MarketBrew.ai, April 2025
Step 2: Read The Realignment Strategy
The tool provides a realignment strategy if needed. This provides a clear roadmap for adjusting your content to better align with the AI’s interpretation of relevance.
Example: If your page is semantically distant from the top embedding cluster, the realignment strategy will suggest changes, such as reworking your content or shifting focus.
Screenshot from MarketBrew.ai, April 2025
Screenshot from MarketBrew.ai, April 2025
Step 3: Test New Changes
Use the “Test New Content” feature to check how well your content now fits the AIO’s top embedding cluster. Iterative testing and refinement are recommended as AI Overviews evolve.
Screenshot by MarketBrew.ai, April 2025
See Your Content Like A Search Engine & Tune It Like A Pro
You’ve just seen under the hood of modern SEO – embeddings, clusters, and AI Overviews. These aren’t abstract theories. They’re the same core systems that Google uses to determine what ranks.
Think of it like getting access to the Porsche service manual, not just the owner’s guide. Suddenly, you can stop guessing which tweaks matter and start making adjustments that actually move the needle.
At Market Brew, we’ve spent over two decades modeling these algorithms. Tools like the free AI Overviews Visualizer give you that mechanic’s-eye view of how search engines interpret your content.
And for teams that want to go further, a paid license unlocks Ranking Blueprints to help track and prioritize which AIO-based metrics most affect your rankings – like cosine similarity and top embedding clusters.
Google’s John Mueller clarifies that automatically changing XML sitemap dates doesn’t boost SEO and could make it harder for Google to find actual content updates.
The “Freshness Signal” Myth Busted
On Reddit’s r/SEO forum, someone asked if competitors ranked better by setting their XML sitemap dates to the current date to send a “freshness signal” to Google.
Mueller’s answer was clear:
“It’s usually a sign they have a broken sitemap generator setup. It has no positive effect. It’s just a lazy setup.”
The discussion shows a common frustration among SEO pros. The original poster was upset after following Google’s rules for 15 years, only to see competitors using “spam tactics” outrank established websites.
When asked about sites using questionable tactics yet still ranking well, Mueller explained that while some “sneaky things” might work briefly, updating sitemap dates isn’t one of them.
Mueller said:
“Setting today’s date in a sitemap file isn’t going to help anyone. It’s just lazy. It makes it harder for search engines to spot truly updated pages. This definitely isn’t working in their favor.”
XML Sitemaps: What Works
XML sitemaps help search engines understand your website structure and when content was last updated. While good sitemaps are essential for SEO, many people misunderstand the impact they have on rankings.
According to Google, the lastmod tag in XML sitemaps should show when a page was truly last updated. When used correctly, this helps search engines know which pages have new content that needs to be recrawled.
Mueller confirms that faking these dates doesn’t help your rankings and may prevent Google from finding your real content updates.
What This Means for Your SEO
Mueller’s comments remind us that while some SEO tactics might seem to improve rankings, correlation isn’t causation.
Sites ranking well despite questionable methods are likely succeeding due to other factors, rather than manipulated sitemap dates.
For website owners and SEO professionals, the advice is:
Keep your XML sitemaps accurate
Only update lastmod dates when you change content
Focus on creating valuable content instead of technical shortcuts
Be patient with ethical SEO strategies – they provide lasting results
It can be frustrating to see competitors seemingly benefit from questionable tactics. However, Mueller suggests these advantages don’t last long and can backfire.
This exchange confirms that Google’s smart algorithms can recognize and eventually ignore artificial attempts to manipulate ranking signals.
Google’s Search Relations team has released a new SEO Office Hours video with Martin Splitt.
He tackles a common problem many website owners face: unwanted noindex tags that keep pages out of search results.
In the video, Splitt helps a user named Balant who couldn’t remove a noindex tag from their website. Balant wanted their page to be public, but the tag prevented this.
Where Unwanted Noindex Tags Come From
Splitt listed several places where unwanted noindex tags might be hiding:
“Make sure that it’s not in the source code, it’s not coming from JavaScript, it’s not coming from a third-party JavaScript.”
Splitt pointed out that A/B testing tools often cause this problem. These tools sometimes add noindex tags to test versions of your pages without you realizing it.
CDN & Cache Problems
If you use a Content Delivery Network (CDN), Splitt warned that old cached versions might still have noindex tags even after you remove them from your site.
Splitt explained:
“If you had a noindex in and you’re using a CDN, it might be that the cache hasn’t updated yet.”
Check Your CMS Settings & Plugins
Splitt explained that your Content Management System (CMS) settings might be adding noindex tags without you knowing.
He said:
“If you’re using a CMS, there might be settings or plugins for SEO, and there might be something like ‘allow search engines to index this content’ or ‘to access this content,’ and you want to make sure that’s set.”
Splitt added that settings labeled as “disallow search engines” should be unchecked if you want your content to appear in search results.
See the full video:
Debugging Process for Persistent Noindex Issues
If you’re dealing with stubborn noindex problems, Splitt suggests checking these places in order:
Check your HTML source code directly
Look at JavaScript files that might add meta tags
Review third-party scripts, especially testing tools
Check if your CDN cache needs updating
Look at your CMS settings and SEO plugins
What This Means For SEO Professionals
Google’s advice shows why thorough technical SEO checks are essential. Modern websites are complex with dynamic content and third-party tools, so finding technical SEO problems takes deeper digging.
SEO professionals should regularly crawl their sites with tools that process JavaScript. This practice provides a deeper understanding of how search engines interpret your pages, going beyond the basic HTML and revealing the true visibility of your content.
Google keeps covering these basic technical issues in its videos, suggesting that even well-designed websites often struggle with indexing problems.
If your pages aren’t showing up in search results, use Google’s URL Inspection tool in the Search Console. This shows you how Google sees your page and whether any noindex tags exist.
Google Search Relations team members recently shared insights about web standards on the Search Off the Record podcast.
Martin Splitt and Gary Illyes explained how these standards are created and why they matter for SEO. Their conversation reveals details about Google’s decisions that affect how we optimize websites.
Why Some Web Protocols Become Standards While Others Don’t
Google has formally standardized robots.txt through the Internet Engineering Task Force (IETF). However, they left the sitemap protocol as an informal standard.
This difference illustrates how Google determines which protocols require official standards.
“With robots.txt, there was a benefit because we knew that different parsers tend to parse robots.txt files differently… With sitemap, it’s like ‘eh’… it’s a simple XML file, and there’s not that much that can go wrong with it.”
This statement from Illyes reveals Google’s priorities. Protocols that confuse platforms receive more attention than those that work well without formal standards.
The Benefits of Protocol Standardization for SEO
The standardization of robots.txt created several clear benefits for SEO:
Consistent implementation: Robots.txt files are now interpreted more consistently across search engines and crawlers.
Open-source resources: “It allowed us to open source our robots.txt parser and then people start building on it,” Illyes noted.
Easier to use: According to Illyes, standardization means “there’s less strain on site owners trying to figure out how to write the damned files.”
These benefits make technical SEO work more straightforward and more effective, especially for teams managing large websites.
Inside the Web Standards Process
The podcast also revealed how web standards are created.
Standards groups, such as the IETF, W3C, and WHATWG, work through open processes that often take years to complete. This slow pace ensures security, clear language, and broad compatibility.
Illyes explained:
“You have to show that the thing you are working on actually works. There’s tons of iteration going on and it makes the process very slow—but for a good reason.”
Both Google engineers emphasized that anyone can participate in these standards processes. This creates opportunities for SEO professionals to help shape the protocols they use on a daily basis.
Security Considerations in Web Standards
Standards also address important security concerns. When developing the robots.txt standard, Google included a 500-kilobyte limit specifically to prevent potential attacks.
Illyes explained:
“When I’m reading a draft, I would look at how I would exploit stuff that the standard is describing.”
This demonstrates how standards establish security boundaries that safeguard both websites and the tools that interact with them.
Why This Matters
For SEO professionals, these insights indicate several practical strategies to consider:
Be precise when creating robots.txt directives, since Google has invested heavily in this protocol.
Use Google’s open-source robots.txt parser to check your work.
Know that sitemaps offer more flexibility with fewer parsing concerns.
Consider joining web standards groups if you want to help shape future protocols.
As search engines continue to prioritize technical quality, understanding the underlying principles behind web protocols becomes increasingly valuable for achieving SEO success.
This conversation shows that even simple technical specifications involve complex considerations around security, consistency, and ease of use, all factors that directly impact SEO performance.
How can you structure content so that AI pulls your site into the response?
Do you really need to change your SEO strategy?
For years, SEO teams followed a familiar SEO playbook:
Optimize upper-funnel content to capture awareness,
mid-funnel content to drive consideration,
lower-funnel content to convert.
One page, one keyword, one intent.
But with the rise of ChatGPT, Perplexity, Copilot, Gemini, and now Google’s AI Mode, that linear model is increasingly outdated.
So, how do you move forward and keep your visibility high in modern search engine results pages (SERPs)?
We’ve reverse-engineered AI Overviews, so you don’t have to. Let’s dive in.
What We’ve Discovered Through Reverse Engineering Google’s AI Overviews (AIO)
From what we’re seeing across client industries and in how AI-driven results behave, the traditional funnel model – the idea of users moving cleanly from awareness to consideration to conversion – feels increasingly out of step with how people actually search.
How Today’s Search Users Actually Search
Today’s users jump between channels, devices, and questions.
They skim, abandon, revisit, and decide faster than ever.
AI Overviews don’t follow a tidy funnel because most people don’t either.
They surface multiple types of information at once, not because it’s smarter SEO, but because it’s closer to how real decisions get made.
AIOs & AI Mode Aren’t Just Answering Queries – They’re Expanding Them
Traditionally, SEO strategy followed a structured framework. Take a travel-related topic, for example:
Informational (Upper-Funnel) – “How to plan a cruise?”
Commercial (Mid-Funnel) – “Best cruise lines for families”
Transactional (lower-Funnel) – “Find Best Alaska Cruise Deals”
However, AI Overviews don’t stick to that structure.
Instead, they blend multiple layers of intent into a single, comprehensive response.
How AI Overviews Answer & Expand Search Queries
Let’s stay with the travel theme. A search for “Mediterranean cruise” might return an AI Overview that includes:
Best Time to go (Informational).
Booking Your Cruise (Commercial).
Cruise Lines (Navigational).
AI Mode Example for ‘Mediterranean Cruise’
What’s Happening Here?
In this case, Google isn’t just answering the query.
It anticipates what the user will want to know next, acting more like a digital concierge than a traditional search engine.
We started noticing this behavior showing up more often, so we wanted to see how common it actually is.
To get a clearer picture, we pulled 200 cruise-related informational queries from SEMrush and ran them through our custom-built AI SERP scraper. The goal was to see how often these queries triggered AI Overviews, and what kind of intent those Overviews covered.
The patterns were hard to miss:
88% of those queries triggered an AI Overview
More than half didn’t just answer the initial question.
52% mixed in other layers of intent, like brand suggestions, booking options, or comparisons, right alongside the basic information someone might’ve been looking for.
Using a different query related to Mediterranean Cruises, the AIO response acts as a travel agent, guiding the user on topics like:
How to fly,
Destinations with region,
Cruise prices,
Cruise lines that sail to that destination.
While it’s an Information non-brand search query, the AIO response is lower-funnel as well.
Again, less than half of the queries were matched intent.
Here are some examples of queries that were identified as Informational and provided only the top-of-funnel response without driving the user further down the funnel.
The Verdict
Even when someone asks a simple, top-of-funnel question, AI is already steering them toward what to do next, whether that’s comparing prices, picking a provider, or booking a trip.
What Does This Mean for SEO Strategies Moving Forward?
If AI Overviews and AI Mode are blending intent types, content, and SEO strategies need to catch up:
It’s no longer enough to rank for high-volume informational keywords. If your content doesn’t address multiple layers of intent, AI will fill the gaps with someone else’s content.
SEO teams need to analyze how AI handles their most important queries. What related questions is it pulling in? Are those answers coming from your site or your competitors?
Think beyond keyword volume. Long-tail queries may have lower search traffic, but they often align better with AI-cited content. Structure your pages with clear headings, bullets, and concise, helpful language—that’s what AI models prefer to surface.
The Future of SEO in an AI World: Hybrid Intent Optimization
The fundamentals of technical and on-page SEO still matter. But if your content is still built around single keywords and single intent types, you’re likely to lose visibility as AI continues to reshape the SERP.
The brands that adapt to this shift by creating content that mirrors the blended, fast-moving behavior of actual users are the ones that will continue to own key moments across the funnel, even as the funnel itself evolves.
AsAItransformssearchbehavior,it‘scrucialtoadaptyourSEOstrategiesaccordingly.AtDAC,wespecializeinaligningyourcontentwiththelatestsearchtrendstoenhancevisibilityandengagement.Reachouttoustodaytofuture-proofyour strategy with our award-winning TotalSERP approachandstayaheadintheevolvingdigitallandscape.
https://www.dacgroup.com/” class=”btn-learn-more button-green medium-size”>Optimize Your SEO For AI Search, Now
Image Credits
Featured Image: Image by DAC. Used with permission.
In-Post Image: Images by DAC. Used with permission.
Site migration issues happen. You plan, create a staging site, and then when the site goes live, there’s bound to be something wrong.
Quality assurance gets thrust into overdrive the moment that migrations are complete.
You sift through thousands of pages, metadata, and more to fix any problems before someone else notices.
It’s a lot of work and time-consuming to feel confident that a site migration is complete without issues.
But, I’m going to show you how to identify migration issues quickly using Google Sheets and AI. You still have a lot to do (migration experts, rejoice!), but this script is going to help you:
Compare old and new ScreamingFrog crawls.
Identify immediate issues that you need to resolve.
SEOs have their own strategies and practices that they follow, and this script is going to allow you to QA migrations quickly based on your own requirements.
You can adapt the script below to make this work for you, whether you’re working on a small local business site or an enterprise.
Setting Everything Up With Screaming Frog And Google Sheets
I’m using Screaming Frog for this example because it makes it easy for me to export data for both sites.
We’re going to assume the following:
Your first version is your live website, which we’ll call the Old Crawl.
Your second version is your new site on a staging environment, which we’ll call New Crawl.
You’re going to create a Google Sheets with the following Sheets:
Overview.
Old Crawl.
New Crawl.
Once your Sheet is set up properly, run your ScreamingFrog scan using any settings that you like.
You’ll run the scan for your Old and New Crawl and then inmport the data to the Old Crawl and New Crawl tabs in your Sheets.
Your sheets will look something like this:
ScreamingFrog Export Crawl Results (Screenshot of Google Sheet, March 2025)
The New Crawl will look very similar.
Once you fill in both the New and Old Crawl sheets, you’ll need to populate your Overview sheet.
The table that you create in this sheet should contain the following columns:
Existing (old) URL.
New URL.
Status Code.
Indexability.
Title 1.
Meta Description 1.
H1-1.
H2-1.
Column 3.
Column 4.
Your Overview sheet will look something like this:
Migration QA Overview Sheet (Screenshot of Google Sheet, March 2025)
Once you have your sheets set up, it’s time to put your favorite AI to work to compare your data.
I used ChatGPT, but you can use any AI you like. I’m sure Claude, Deepseek, or Gemini would do equally as well as long as you use similar prompts.
Prompts To Create Your Google Sheets Data
You can fill in your Google Sheet formulas by hand if you’re a formula guru, but it’s easier to let AI do it for you since we’re making basic comparisons.
Remember, the Old Crawl is the live site, and the New Crawl is my staging site.
Now, go to your AI tool and prompt it with the following:
I need a Google Sheets formula that compares values between two sheets: "Old Crawl" and "New Crawl." The formula should:
Look up a value in column A of "Old Crawl" using the value in column A of the current sheet.
Look up a value in column A of "New Crawl" using the value in column B of the current sheet.
Find the corresponding column in both sheets by matching the column header in row 1 with the current column header.
If the values match, return "Pass".
If they don't match, return "Error (old<>new)" with the differing values shown.
Use TEXTJOIN("<>", TRUE, ...) to format the error message.
Ensure compatibility with Google Sheets by specifying explicit ranges instead of full-column references.
You can adjust these prompt points on your own.
For example, you can change “Old Crawl” to “Live Site,” but be sure that the sheet names match up properly.
ChatGPT generated code for me that looks something like this:
You can use these basic formulas to start comparing rows by pasting the formula in row 2.
Adding the formula is as simple as double-clicking the field and pasting it in.
I know that you’ll want to make this a little more complex. You can do a lot of things with Google Sheets and formulas, so tweak things as needed.
Ideas For Expanding Your Migration Sheet
Your formulas will depend on the settings of your Screaming Frog crawl, but here are a few that I think will work well:
Create a function to compare all of the status codes between the Old Crawl and New Crawl to identify key issues that exist. For example, if a page has anything but a 200 code, you can highlight the issue to fix it quickly.
Add a formula to highlight metadata that is too long or short, so that you can add it to your task list for when the audit is over.
Create a function to monitor Response Time between both the Old and New Crawl so that you can identify any issues that the new crawl may have or report speed increases if switching to a new host or server.
Create another function to compare the URL structure of each URL. You might compare trailing slashes, structure and more.
Develop a new function for Inlinks to be sure that no internal links were lost in the migration. You can also check external links using the same concept.
Google’s Martin Splitt recently shared insights on how JavaScript mistakes can hurt a website’s search performance.
His talk comes as Google Search Advocate John Mueller also urges SEO pros to learn more about modern client-side technologies.
Mistake 1: Rendered HTML vs. Source HTML
During the SEO for Paws Conference, a live-streamed fundraiser by Anton Shulke, Splitt drew attention to a trend he’s noticing.
Many SEO professionals still focus on the website’s original source code even though Google uses the rendered HTML for indexing. Rendered HTML is what you see after JavaScript has finished running.
Splitt explains:
“A lot of people are still looking at view source. That is not what we use for indexing. We use the rendered HTML.”
This is important because JavaScript can change pages by removing or adding content. Understanding this can help explain some SEO issues.
Mistake 2: Error Pages Being Indexed
Splitt pointed out a common error with single-page applications and JavaScript-heavy sites: they often return a 200 OK status for error pages.
This happens because the server sends a 200 response before the JavaScript checks if the page exists.
Splitt explains:
“Instead of responding with 404, it just responds with 200 … always showing a page based on the JavaScript execution.”
When error pages get a 200 code, Google indexes them like normal pages, hurting your SEO.
Splitt advises checking server settings to handle errors properly, even when using client-side rendering.
Mistake 3: Geolocation Request Issue
Another problem arises when sites ask users for location or other permissions.
Splitt says Googlebot will always refuse the request if a site relies on geolocation (or similar requests) without a backup plan.
Splitt explains:
“Googlebot does not say yes on that popup. It says no on all these requests … so if you request geolocation, Googlebot says no.”
The page can appear blank to Googlebot without alternative content, meaning nothing is indexed. This can turn into a grave SEO mistake.
How to Debug JavaScript for SEO
Splitt shared a few steps to help diagnose and fix JavaScript issues:
Start with Search Console: Use the URL Inspection tool to view the rendered HTML.
Check the Content: Verify if the expected content is there.
Review HTTP Codes: Look at the status codes in the “More info” > “Resources” section.
Use Developer Tools: Open your browser’s developer tools. Check the “initiator” column in the Network tab to see which JavaScript added specific content.
Splitt adds:
“The initiator is what loaded it. If it’s injected by JavaScript, you can see which part of the code did it.”
Following these steps can help you find the problem areas and work with your developers to fix them.
See Splitt’s full talk in the recording below:
A Shift in SEO Skills
Splitt’s advice fits with Mueller’s call for SEOs to broaden their skill set.
Mueller recently suggested that SEO professionals learn about client-side frameworks, responsive design, and AI tools.
Mueller stated:
“If you work in SEO, consider where your work currently fits in … if your focus was ‘SEO at server level,’ consider that the slice has shrunken.”
Modern JavaScript techniques create new challenges that old SEO methods cannot solve alone. Splitt’s real-world examples show why understanding these modern web practices is now critical.
What This Means For SEO Professionals
Both Google Advocates point to a clear trend: SEO now requires more technical skills. As companies look for professionals who can blend SEO and web development, the demand for these modern skills is growing.
To keep up, SEO pros should:
Learn How JavaScript Affects Indexing: Know the difference between source and rendered HTML.
Master Developer Tools: Use tools like Search Console and browser developer tools to spot issues.
Collaborate with Developers: Work together to build sites that serve users and search engines well.
Broaden Your Skillset: Add client-side techniques to your traditional SEO toolkit.
Looking Ahead
As the web evolves, so must the skills of SEO professionals. However, leveling up your knowledge doesn’t have to be intimidating.
This fresh look at JavaScript’s role in SEO shows that even simple changes can have a big impact.
