Google Formalizes Decade-Old Faceted Navigation Guidelines via @sejournal, @MattGSouthern

Google has updated its guidelines on faceted navigation by turning an old blog post into an official help document.

What started as a blog post in 2014 is now official technical documentation.

This change reflects the complexity of ecommerce and content-heavy websites, as many sites adopt advanced filtering systems for larger catalogs.

Faceted Navigation Issues

Ever used filters on an e-commerce site to narrow down products by size, color, and price?

That’s faceted navigation – the system allowing users to refine search results using multiple filters simultaneously.

While this feature is vital for users, it can create challenges for search engines, prompting Google to release new official documentation on managing these systems.

Modern Challenges

The challenge with faceted navigation lies in the mathematics of combinations: each additional filter option multiplies the potential URLs a search engine might need to crawl.

For example, a simple product page with options for size (5 choices), color (10 choices), and price range (6 ranges) could generate 300 unique URLs – for just one product.

According to Google Analyst Gary Illyes, this multiplication effect makes faceted navigation the leading cause of overcrawling issues reported by website owners.

The impact includes:

• Wasting Server Resources: Many websites use too much computing power on unnecessary URL combinations.
• Inefficient Crawl Budget: Crawlers may take longer to find important new content because they are busy with faceted navigation.
• Weakening SEO Performance: Having several URLs for the same content can hurt a website’s SEO.

What’s Changed?

The new guidance is similar to the 2014 blog post, but it includes some important updates:

1. Focus on Performance: Google now clearly warns about the costs of using computing resources.
2. Clear Implementation Options: The documentation gives straightforward paths for different types of websites.
3. Updated Technical Recommendations: Suggestions now account for single-page applications and modern SEO practices.

Implementation Guide

For SEO professionals managing sites with faceted navigation, Google now recommends a two-track approach:

Non-Critical Facets:

• Block via robots.txt
• Use URL fragments (#)
• Implement consistent rel=”nofollow” attributes

Business-Critical Facets:

• Maintain standardized parameter formats
• Implement proper 404 handling
• Use strategic canonical tags

Looking Ahead

This documentation update suggests Google is preparing for increasingly complex website architectures.

SEO teams should evaluate their current faceted navigation against these guidelines to ensure optimal crawling efficiency and indexing performance.

Featured Image: Shutterstock/kenchiro168

Google Warns: Beware Of Fake Googlebot Traffic via @sejournal, @MattGSouthern

Google’s Developer Advocate, Martin Splitt, warns website owners to be cautious of traffic that appears to come from Googlebot. Many requests pretending to be Googlebot are actually from third-party scrapers.

He shared this in the latest episode of Google’s SEO Made Easy series, emphasizing that “not everyone who claims to be Googlebot actually is Googlebot.”

Why does this matter?

Fake crawlers can distort analytics, consume resources, and make it difficult to assess your site’s performance accurately.

Here’s how to distinguish between legitimate Googlebot traffic and fake crawler activity.

Googlebot Verification Methods

You can distinguish real Googlebot traffic from fake crawlers by looking at overall traffic patterns rather than unusual requests.

Real Googlebot traffic tends to have consistent request frequency, timing, and behavior.

If you suspect fake Googlebot activity, Splitt advises using the following Google tools to verify it:

URL Inspection Tool (Search Console)

• Finding specific content in the rendered HTML confirms that Googlebot can successfully access the page.
• Provides live testing capability to verify current access status.

Rich Results Test

• Acts as an alternative verification method for Googlebot access
• Shows how Googlebot renders the page
• Can be used even without Search Console access

Crawl Stats Report

• Shows detailed server response data specifically from verified Googlebot requests
• Helps identify patterns in legitimate Googlebot behavior

There’s a key limitation worth noting: These tools verify what real Googlebot sees and does, but they don’t directly identify impersonators in your server logs.

To fully protect against fake Googlebots, you would need to:

• Compare server logs against Google’s official IP ranges
• Implement reverse DNS lookup verification
• Use the tools above to establish baseline legitimate Googlebot behavior

Monitoring Server Responses

Splitt also stressed the importance of monitoring server responses to crawl requests, particularly:

• 500-series errors
• Fetch errors
• Timeouts
• DNS problems

These issues can significantly impact crawling efficiency and search visibility for larger websites hosting millions of pages.

Splitt says:

“Pay attention to the responses your server gave to Googlebot, especially a high number of 500 responses, fetch errors, timeouts, DNS problems, and other things.”

He noted that while some errors are transient, persistent issues “might want to investigate further.”

Splitt suggested using server log analysis to make a more sophisticated diagnosis, though he acknowledged that it’s “not a basic thing to do.”

However, he emphasized its value, noting that “looking at your web server logs… is a powerful way to get a better understanding of what’s happening on your server.”

Potential Impact

Beyond security, fake Googlebot traffic can impact website performance and SEO efforts.

Splitt emphasized that website accessibility in a browser doesn’t guarantee Googlebot access, citing various potential barriers, including:

• Robots.txt restrictions
• Firewall configurations
• Bot protection systems
• Network routing issues

Looking Ahead

Fake Googlebot traffic can be annoying, but Splitt says you shouldn’t worry too much about rare cases.

Suppose fake crawler activity becomes a problem or uses too much server power. In that case, you can take steps like limiting the rate of requests, blocking specific IP addresses, or using better bot detection methods.

For more on this issue, see the full video below:

Featured Image: eamesBot/Shutterstock

Google Confirms Search Indexing Issue via @sejournal, @MattGSouthern

Google confirms widespread search indexing delays affecting a “small number” of websites.

• Google Search is experiencing indexing delays.
• The company says only a small number of sites are affected and is investigating the root cause.
• Google will provide an update on the situation soon.

Google: Focus On Field Data For Core Web Vitals via @sejournal, @MattGSouthern

Google stresses the importance of using actual user data to assess Core Web Vitals instead of relying only on lab data from tools like PageSpeed Insights (PSI) and Lighthouse.

This reminder comes as the company prepares to update the throttling settings in PSI. These updates are expected to increase the performance scores of websites in Lighthouse.

Field Data vs. Lab Data

Core Web Vitals measure a website’s performance in terms of loading speed, interactivity, and visual stability from the user’s perspective.

Field data shows users’ actual experiences, while lab data comes from tests done in controlled environments using tools like Lighthouse.

Barry Pollard, a Web Performance Developer Advocate at Google, recently emphasized focusing on field data.

In a LinkedIn post, he stated:

“You should concentrate on your field Core Web Vitals (the top part of PageSpeed Insights), and only use the lab Lighthouse Score as a very rough guide of whether Lighthouse has recommendations to improve performance or not…

The Lighthouse Score is best for comparing two tests made on the same Lighthouse (e.g. to test and compare fixes).

Performance IS—and hence LH Scores also ARE—highly variable. LH is particularly affected by where it is run from (PSI, DevTools, CI…), but also on the lots of other factors.

Lighthouse is a GREAT tool but it also can only test some things, under certain conditions.

So while it’s great to see people interested in improving webperf, make sure you’re doing just that (improve performance) and not just improving the score”

Upcoming Changes To PageSpeed Insights

Pollard discussed user concerns about PageSpeed Insights’s slow servers, which can cause Lighthouse tests to take longer than expected.

To fix this, Google is changing the throttling settings in PageSpeed Insights, which should lead to better performance scores when the update is released in the coming weeks.

These changes will affect both the web interface and the API but will not impact other versions of Lighthouse.

However, Pollard  reminds users that  “a score of 100 doesn’t mean perfect; it just means Lighthouse can’t help anymore.”

Goodhart’s Law & Web Performance

Pollard referenced Goodhart’s Law, which says that when a measure becomes a goal, it stops being a good measure.

In the web performance context, focusing only on improving Lighthouse scores may not improve actual user experience.

Lighthouse is a helpful tool, but it can only assess certain aspects of performance in specific situations.

Alon Kochba, Web Performance and Software Engineer at Wix, added context to the update, stating:

“Lighthouse scores may not be the most important – but this is a big deal for Lighthouse scores in PageSpeed Insights.

4x -> 1.2x CPU throttling for Mobile device simulation, which was way off for quite a while.”

Lighthouse scores may not be the most important – but this is a big deal for Lighthouse scores in PageSpeed Insights.

4x -> 1.2x CPU throttling for Mobile device simulation, which was way off for quite a while. https://t.co/ZLrwsDQGmO

— Alon Kochba (@alonkochba) December 6, 2024

Key Takeaway: Prioritize User Experience

As the update rolls out, website owners and developers should focus on user experience using field data for Core Web Vitals.

While Lighthouse scores can help find areas for improvement, they shouldn’t be the only goal.

Google encourages creating websites that load quickly, respond well, and are visually stable.

Featured Image: GoodStudio/Shutterstock

Google Uses About 40 Signals To Determine Canonical URLs via @sejournal, @MattGSouthern

In a recent episode of Google’s Search Off the Record podcast, Allan Scott from the “Dups” team explained how Google decides which URL to consider as the main one when there are duplicate pages.

He revealed that Google looks at about 40 different signals to pick the main URL from a group of similar pages.

Around 40 Signals For Canonical URL Selection

Duplicate content is a common problem for search engines because many websites have multiple pages with the same or similar content.

To solve this, Google uses a process called canonicalization. This process allows Google to pick one URL as the main version to index and show in search results.

Google has discussed the importance of using signals like rel=”canonical” tags, sitemaps, and 301 redirects for canonicalization. However, the number of signals involved in this process is more than you may expect.

Scott revealed during the podcast:

“I’m not sure what the exact number is right now because it goes up and down, but I suspect it’s somewhere in the neighborhood of 40.”

Some of the known signals mentioned include:

1. rel=”canonical” tags
2. 301 redirects
3. HTTPS vs. HTTP
4. Sitemaps
5. Internal linking
6. URL length

The weight and importance of each signal may vary, and some signals, like rel=”canonical” tags, can influence both the clustering and canonicalization process.

Balancing Signals

With so many signals at play, Allan acknowledged the challenges in determining the canonical URL when signals conflict.

He stated:

“If your signals conflict with each other, what’s going to happen is the system will start falling back on lesser signals.”

This means that while strong signals like rel=”canonical” tags and 301 redirects are crucial, other factors can come into play when these signals are unclear or contradictory.

As a result, Google’s canonicalization process involves a delicate balancing act to determine the most appropriate canonical URL.

Best Practices For Canonicalization

Clear signals help Google identify the preferred canonical URL.

Best practices include:

1. Use rel=”canonical” tags correctly.
2. Implement 301 redirects for permanently moved content.
3. Ensure HTTPS versions of pages are accessible and linked.
4. Submit sitemaps with preferred canonical URLs.
5. Keep internal linking consistent.

These signals help Google find the correct canonical URLs, improving your site’s crawling, indexing, and search visibility.

Mistakes To Avoid

Here are a few common mistakes to watch out for.

1. Incorrect or conflicting canonical tags:

• Pointing to non-existent or 404 pages
• Multiple canonical tags with different URLs on one page
• Pointing to a different domain entirely

Fix: Double-check canonical tags, use only one per page, and use absolute URLs.

2. Canonical chains or loops

When Page A points to Page B as canonical, but Page B points back to A or another page, creating a loop.

Fix: Ensure canonical URLs always point to the final, preferred version of the page.

3. Using noindex and canonical tags together

Sending mixed signals to search engines. Noindex means don’t index the page at all, making canonicals irrelevant.

Fix: Use canonical tags for consolidation and noindex for exclusion.

4. Canonicalizing to redirect or noindex pages

Pointing canonicals to redirected or noindex pages confuses search engines.

Fix: Canonical URLs should be 200 status and indexable.

5. Ignoring case sensitivity

Inconsistent URL casing can cause duplicate content issues.

Fix: Keep URL and canonical tag casing consistent.

6. Overlooking pagination and parameters

Paginated content and parameter-heavy URLs can cause duplication if mishandled.

Fix: Use canonical tags pointing to the first page or “View All” for pagination, and keep parameters consistent.

Key Takeaways

It’s unlikely the complete list of 40+ signals used to determine canonical URLs will be made publicly available.

However, this was still an insightful discussion worth highlighting.

Here are the key takeaways:

• Google uses approximately 40 different signals to determine canonical URLs, with rel=”canonical” tags and 301 redirects being among the strongest indicators
• When signals conflict, Google falls back on secondary signals to make its determination
• Clear, consistent implementation of canonicalization signals (tags, redirects, sitemaps, internal linking) is crucial
• Common mistakes like canonical chains, mixed signals, or incorrect implementations can confuse search engines

Hear the full discussion in the video below:

Featured Image: chatiyanon/Shutterstock

Google Warns Of Duplicate Content “Black Holes” Caused By Error Pages via @sejournal, @MattGSouthern

Google’s “Search Off the Record” podcast recently highlighted an SEO issue that can make web pages disappear from search results.

In the latest episode, Google Search team member Allan Scott discussed “marauding black holes” formed by grouping similar-looking error pages.

Google’s system can accidentally cluster error pages that look alike, causing regular pages to get included in these groups.

This means Google may not crawl these pages again, which can lead to them being de-indexed, even after fixing the errors.

The podcast explained how this happens, its effects on search traffic, and how website owners can keep their pages from getting lost.

How Google Handles Duplicate Content

To understand content black holes, you must first know how Google handles duplicate content.

Scott explains this happens in two steps:

1. Clustering: Google groups pages that have the same or very similar content.
2. Canonicalization: Google then chooses the best URL from each group.

After clustering, Google stops re-crawling these pages. This saves resources and avoids unnecessary indexing of duplicate content.

How Error Pages Create Black Holes

The black hole problem happens when error pages group together because they have similar content, such as generic “Page Not Found” messages. Regular pages with occasional errors or temporary outages can get stuck in these error clusters.

The duplication system prevents the re-crawling of pages within a cluster. This makes it hard for mistakenly grouped pages to escape the “black hole,” even after fixing the initial errors. As a result, these pages can get de-indexed, leading to a loss of organic search traffic.

Scott explained:

“Only the things that are very towards the top of the cluster are likely to get back out. Where this really worries me is sites with transient errors… If those fail to fetch, they might break your render, in which case we’ll look at your page, and we’ll think it’s broken.”

How To Avoid Black Holes

To avoid problems with duplicate content black holes, Scott shared the following advice:

1. Use the Right HTTP Status Codes: For error pages, use proper status codes (like 404, 403, and 503) instead of a 200 OK status. Only pages marked as 200 OK may be grouped together.
2. Create Unique Content for Custom Error Pages: If you have custom error pages that use a 200 OK status (common in single-page apps), make sure these pages contain specific content to prevent grouping. For example, include the error code and name in the text.
3. Caution with Noindex Tags: Do not use noindex tags on error pages unless you want them permanently removed from search results. This tag strongly indicates that you want the pages removed, more so than using error status codes.

Following these tips can help ensure regular pages aren’t accidentally mixed with error pages, keeping them in Google’s index.

Regularly checking your site’s crawl coverage and indexation can help catch duplication issues early.

In Summary

Google’s “Search Off the Record” podcast highlighted a potential SEO issue where error pages can be seen as duplicate content. This can cause regular pages to be grouped with errors and removed from Google’s index, even if the errors are fixed.

To prevent duplicate content issues, website owners should:

1. Use the correct HTTP status codes for error pages.
2. Ensure custom error pages have unique content.
3. Monitor their site’s crawl coverage and indexation.

Following technical SEO best practices is essential for maintaining strong search performance, as emphasized by Google’s Search team.

Hear the full discussion in the video below:

Featured Image: Nazarii_Neshcherenskyi/Shutterstock

Google On Robots.txt: When To Use Noindex vs. Disallow via @sejournal, @MattGSouthern

In a recent YouTube video, Google’s Martin Splitt explained the differences between the “noindex” tag in robots meta tags and the “disallow” command in robots.txt files.

Splitt, a Developer Advocate at Google, pointed out that both methods help manage how search engine crawlers work with a website.

However, they have different purposes and shouldn’t be used in place of each other.

When To Use Noindex

The “noindex” directive tells search engines not to include a specific page in their search results. You can add this instruction in the HTML head section using the robots meta tag or the X-Robots HTTP header.

Use “noindex” when you want to keep a page from showing up in search results but still allow search engines to read the page’s content. This is helpful for pages that users can see but that you don’t want search engines to display, like thank-you pages or internal search result pages.

When To Use Disallow

The “disallow” directive in a website’s robots.txt file stops search engine crawlers from accessing specific URLs or patterns. When a page is disallowed, search engines will not crawl or index its content.

Splitt advises using “disallow” when you want to block search engines completely from retrieving or processing a page. This is suitable for sensitive information, like private user data, or for pages that aren’t relevant to search engines.

Common Mistakes to Avoid

One common mistake website owners make is using “noindex” and “disallow” for the same page. Splitt advises against this because it can cause problems.

If a page is disallowed in the robots.txt file, search engines cannot see the “noindex” command in the page’s meta tag or X-Robots header. As a result, the page might still get indexed, but with limited information.

To stop a page from appearing in search results, Splitt recommends using the “noindex” command without disallowing the page in the robots.txt file.

Google provides a robots.txt report in Google Search Console to test and monitor how robots.txt files affect search engine indexing.

Why This Matters

Understanding the proper use of “noindex” and “disallow” directives is essential for SEO professionals.

Following Google’s advice and using the available testing tools will help ensure your content appears in search results as intended.

See the full video below:

Featured Image: Asier Romero/Shutterstock

Google: Host Resources On Different Hostname To Save Crawl Budget via @sejournal, @MattGSouthern

Google Search Central has launched a new series called “Crawling December” to provide insights into how Googlebot crawls and indexes webpages.

Google will publish a new article each week this month exploring various aspects of the crawling process that are not often discussed but can significantly impact website crawling.

The first post in the series covers the basics of crawling and sheds light on essential yet lesser-known details about how Googlebot handles page resources and manages crawl budgets.

Crawling Basics

Today’s websites are complex due to advanced JavaScript and CSS, making them harder to crawl than old HTML-only pages. Googlebot works like a web browser but on a different schedule.

When Googlebot visits a webpage, it first downloads the HTML from the main URL, which may link to JavaScript, CSS, images, and videos. Then, Google’s Web Rendering Service (WRS) uses Googlebot to download these resources to create the final page view.

Here are the steps in order:

1. Initial HTML download
2. Processing by the Web Rendering Service
3. Resource fetching
4. Final page construction

Crawl Budget Management

Crawling extra resources can reduce the main website’s crawl budget. To help with this, Google says that “WRS tries to cache every resource (JavaScript and CSS) used in the pages it renders.”

It’s important to note that the WRS cache lasts up to 30 days and is not influenced by the HTTP caching rules set by developers.

This caching strategy helps to save a site’s crawl budget.

Recommendations

This post gives site owners tips on how to optimize their crawl budget:

1. Reduce Resource Use: Use fewer resources to create a good user experience. This helps save crawl budget when rendering a page.
2. Host Resources Separately: Place resources on a different hostname, like a CDN or subdomain. This can help shift the crawl budget burden away from your main site.
3. Use Cache-Busting Parameters Wisely: Be careful with cache-busting parameters. Changing resource URLs can make Google recheck them, even if the content is the same. This can waste your crawl budget.

Also, Google warns that blocking resource crawling with robots.txt can be risky.

If Google can’t access a necessary resource for rendering, it may have trouble getting the page content and ranking it properly.

Monitoring Tools

The Search Central team says the best way to see what resources Googlebot is crawling is by checking a site’s raw access logs.

You can identify Googlebot by its IP address using the ranges published in Google’s developer documentation.

Why This Matters

This post clarifies three key points that impact how Google finds and processes your site’s content:

• Resource management directly affects your crawl budget, so hosting scripts and styles on CDNs can help preserve it.
• Google caches resources for 30 days regardless of your HTTP cache settings, which helps conserve your crawl budget.
• Blocking critical resources in robots.txt can backfire by preventing Google from properly rendering your pages.

Understanding these mechanics helps SEOs and developers make better decisions about resource hosting and accessibility – choices that directly impact how well Google can crawl and index their sites.

Featured Image: ArtemisDiana/Shutterstock