Google Revamps Entire Crawler Documentation via @sejournal, @martinibuster

Google has launched a major revamp of its Crawler documentation, shrinking the main overview page and splitting content into three new, more focused pages.  Although the changelog downplays the changes there is an entirely new section and basically a rewrite of the entire crawler overview page. The additional pages allows Google to increase the information density of all the crawler pages and improves topical coverage.

What Changed?

Google’s documentation changelog notes two changes but there is actually a lot more.

Here are some of the changes:

  • Added an updated user agent string for the GoogleProducer crawler
  • Added content encoding information
  • Added a new section about technical properties

The technical properties section contains entirely new information that didn’t previously exist. There are no changes to the crawler behavior, but by creating three topically specific pages Google is able to add more information to the crawler overview page while simultaneously making it smaller.

This is the new information about content encoding (compression):

“Google’s crawlers and fetchers support the following content encodings (compressions): gzip, deflate, and Brotli (br). The content encodings supported by each Google user agent is advertised in the Accept-Encoding header of each request they make. For example, Accept-Encoding: gzip, deflate, br.”

There is additional information about crawling over HTTP/1.1 and HTTP/2, plus a statement about their goal being to crawl as many pages as possible without impacting the website server.

What Is The Goal Of The Revamp?

The change to the documentation was due to the fact that the overview page had become large. Additional crawler information would make the overview page even larger. A decision was made to break the page into three subtopics so that the specific crawler content could continue to grow and making room for more general information on the overviews page. Spinning off subtopics into their own pages is a brilliant solution to the problem of how best to serve users.

This is how the documentation changelog explains the change:

“The documentation grew very long which limited our ability to extend the content about our crawlers and user-triggered fetchers.

…Reorganized the documentation for Google’s crawlers and user-triggered fetchers. We also added explicit notes about what product each crawler affects, and added a robots.txt snippet for each crawler to demonstrate how to use the user agent tokens. There were no meaningful changes to the content otherwise.”

The changelog downplays the changes by describing them as a reorganization because the crawler overview is substantially rewritten, in addition to the creation of three brand new pages.

While the content remains substantially the same, the division of it into sub-topics makes it easier for Google to add more content to the new pages without continuing to grow the original page. The original page, called Overview of Google crawlers and fetchers (user agents), is now truly an overview with more granular content moved to standalone pages.

Google published three new pages:

  1. Common crawlers
  2. Special-case crawlers
  3. User-triggered fetchers

1. Common Crawlers

As it says on the title, these are common crawlers, some of which are associated with GoogleBot, including the Google-InspectionTool, which uses the GoogleBot user agent. All of the bots listed on this page obey the robots.txt rules.

These are the documented Google crawlers:

  • Googlebot
  • Googlebot Image
  • Googlebot Video
  • Googlebot News
  • Google StoreBot
  • Google-InspectionTool
  • GoogleOther
  • GoogleOther-Image
  • GoogleOther-Video
  • Google-CloudVertexBot
  • Google-Extended

3. Special-Case Crawlers

These are crawlers that are associated with specific products and are crawled by agreement with users of those products and operate from IP addresses that are distinct from the GoogleBot crawler IP addresses.

List of Special-Case Crawlers:

  • AdSense
    User Agent for Robots.txt: Mediapartners-Google
  • AdsBot
    User Agent for Robots.txt: AdsBot-Google
  • AdsBot Mobile Web
    User Agent for Robots.txt: AdsBot-Google-Mobile
  • APIs-Google
    User Agent for Robots.txt: APIs-Google
  • Google-Safety
    User Agent for Robots.txt: Google-Safety

3. User-Triggered Fetchers

The User-triggered Fetchers page covers bots that are activated by user request, explained like this:

“User-triggered fetchers are initiated by users to perform a fetching function within a Google product. For example, Google Site Verifier acts on a user’s request, or a site hosted on Google Cloud (GCP) has a feature that allows the site’s users to retrieve an external RSS feed. Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules. The general technical properties of Google’s crawlers also apply to the user-triggered fetchers.”

The documentation covers the following bots:

  • Feedfetcher
  • Google Publisher Center
  • Google Read Aloud
  • Google Site Verifier

Takeaway:

Google’s crawler overview page became overly comprehensive and possibly less useful because people don’t always need a comprehensive page, they’re just interested in specific information. The overview page is less specific but also easier to understand. It now serves as an entry point where users can drill down to more specific subtopics related to the three kinds of crawlers.

This change offers insights into how to freshen up a page that might be underperforming because it has become too comprehensive. Breaking out a comprehensive page into standalone pages allows the subtopics to address specific users needs and possibly make them more useful should they rank in the search results.

I would not say that the change reflects anything in Google’s algorithm, it only reflects how Google updated their documentation to make it more useful and set it up for adding even more information.

Read Google’s New Documentation

Overview of Google crawlers and fetchers (user agents)

List of Google’s common crawlers

List of Google’s special-case crawlers

List of Google user-triggered fetchers

Featured Image by Shutterstock/Cast Of Thousands

Page Speed Insights: 6 Powerful Tips To Optimize Your Website via @sejournal, @DebugBear

This post was sponsored by DebugBear. The opinions expressed in this article are the sponsor’s own.

Having a fast website is important not just to provide a great experience for visitors, but also as an SEO ranking factor.

You’ve probably heard of Google’s PageSpeed Insights tool before.

But do you know how to get the most out of PageSpeed Insights? We’ll look at 6 key tips to help you optimize your website performance.

What Is PageSpeed Insights (PSI)?

Website performance has long impacted Google rankings. Accordingly, Google first launched its free PageSpeed Insights tool back in 2010.

PSI is built to help website operators check how fast their website is as well as provide recommendations for how to improve it.

Why Does Page Speed Matter For SEO?

In 2021, Google introduced a new set of website performance metrics, called the Core Web Vitals. The three metrics are:

  • Largest Contentful Paint: how fast does your website load?
  • Cumulative Layout Shift: do page elements move around unexpectedly?
  • Interaction to Next Paint: does the page respond to user input quickly?

A good page experience is rewarded in Google rankings. There’s a “Good” rating threshold for each metric that you need to reach.

Graphic showing Core Web Vitals rating thresholds, September 2024

How To Test Your Website With PageSpeed Insights

Running a performance test with PageSpeed Insights is easy:

  1. Open PageSpeed Insights
  2. Enter your website URL
  3. Click “Analyze”

Test results will appear in just a few seconds. There’s a lot of data, but we’ll explain what it all means next.

Screenshot of test result on PageSpeed Insights, September 2024

1. Understand Where PageSpeed Insights Data Comes From

Each test result on PageSpeed Insights consists of two key sections: “Discover what real users are experiencing” and “Diagnose performance issues”. Each section shows a different type of page speed data.

What Is The Real User Data In PageSpeed Insights?

The real user data in PSI comes from the Chrome User Experience Report (CrUX).

This data is collected from Chrome users on desktop devices and on mobile devices running Android. To contribute to the CrUX report, users need to:

  • Be logged into their Google account
  • Have opted into browser history synchronization
  • Have enabled usage statistics reporting

Wondering if your experiences are included in this real user data? Open the chrome://ukm URL in your Chrome browser and check if metrics collection is enabled.

The real user tells you how fast your website is for actual visitors and how it’s impacting your SEO.

However, the CrUX report also comes with some limitations:

  • Data is always aggregated over a 28-day period, so you won’t immediately see if your website is getting worse
  • You can see how fast your website is, but CrUX does not tell give you any diagnostic data to speed it up
  • Not every page on your website will have CrUX data, as a minimum number of recorded visits has to be reached before Google publishes the data.

You can use a real user monitoring (RUM) tool to get around these limitations. RUM data has several advantages over CrUX data, like instant updates and detailed diagnostics.

Screenshot of a Core Web Vitals trendline in DebugBear real user monitoring, September 2024

What Is The Diagnostic Data In PageSpeed Insights?

While the real user data tells you how well your site is doing, the diagnostic data gives you insight into how to optimize it.

PageSpeed Insights uses Google’s open source Lighthouse tool to test your website and provide a detailed analysis. A Lighthouse test is run in a controlled lab environment, which can means that a lot more information information can be collected compared to real user data.

The lab-test is also run on-demand, and is not subject to the 28-day delay that applies to CrUX data.

At the top of the Lighthouse report Google’s shows an overall Performance score between 0 and 100. This score does not directly impact rankings – Google uses CrUX data for that. However, a good Lighthouse score usually means that your website is also loading quickly for real users.

The Lighthouse score itself determined based on 5 performance metrics:

  • First Contentful Paint: how quickly does the page start loading?
  • Largest Contentful Paint: when does the main page content show up?
  • Total Blocking Time: are user interactions blocked by CPU processing?
  • Cumulative Layout Shift: does content move around after it appears?
  • Speed Index: how quickly does the page content render overall?
Screenshot of performance metrics in PageSpeed Insights, September 2024

Below the overall Lighthouse assessment you can find diagnostic insight that suggests concrete changes you can make to optimize your website.

Each row audits one particular aspect of your performance. For example, if you eliminate render-blocking resources then it will take less time for page content on your website to become visible.

Screenshot of performance diagnostics in PageSpeed Insights, September 2024

2. Use The Score Calculator To See What’s Dragging Your Score Down

If you want to improve your Performance score on PageSpeed Insights, where do you start?

Every Lighthouse report includes a “View Calculator” link that takes you to the Lighthouse Scoring Calculator. This tool tells you how much of the five metrics that Google has measured is contributing to the overall score.

For example, here we can see that the page we’ve tested has a good Cumulative Layout Shift score, while the Largest Contentful Paint receives a poor rating.

We can also see that each metric is assigned a weight. For example, 30% of the Performance score is determined by the subscore for the Total Blocking Time metric.

Screenshot of the Lighthouse Scoring Calculator, September 2024

3. Review Phase Data For The Largest Contentful Paint (LCP) Metric

One of the most insightful audits is often the “Largest Contentful Paint element.”

This audit shows you the largest content element on the page. The LCP metric measures how long it takes after opening the page for this element to become visible. The largest content element can be any type of page content, for example, a heading or an image.

That’s very useful, but Lighthouse actually provides additional insight by breaking the LCP metric down into four phases (also called subparts):

  • Time to First Byte (TTFB): how quickly does the website server provide the HTML document?
  • Load Delay: How soon after loading the document does the LCP image start downloading
  • Load Time: How long does it take to download the LCP image?
  • Render Delay: How soon after loading the LCP resource does the LCP element become visible?

This information will tell you where you need to focus on your optimization.

For example, in the screenshot below, we can see that the LCP image loaded quickly but then wasn’t rendered right away by the browser. That could be because other resources on the page were blocking the page from rendering.

Screenshot of the Lighthouse Largest Contentful Paint element audit, September 2024

Google recently ran an analysis to find out what LCP subparts contribute the most to the overall metric value. They found that server response time and image load delay are the biggest factors in LCP optimization for most websites.

While many website performance recommendations have focused on using compact modern image formats, image load time was found to be a minor factor on most slow websites.

However, you should still check the data for your website to see what optimizations can have the most impact.

4. Performance Score Variability Between Tests: What Does It Mean?

We’ve already seen that the real user CrUX data is aggregated over a 28-day period. Accordingly, its value is stable and only changes very gradually.

But the same can’t be said about the Performance score and other metrics measured in the lab. Testing the same page twice will rarely result in the exact same measurements, and often will show high variation. And if you run Lighthouse with other tools like Chrome DevTools you’re likely to see even bigger differences.

There are many reasons for differences between Lighthouse tests, for example:

  • Differences in server response time
  • Variation in content, for example due to A/B tests or advertisements
  • Differences across test devices and test locations
  • Inaccuracies during data collection

Google has written a detailed guide on Lighthouse variability. You can run tests several times and look at the average to get a more consistent assessment.

Data Accuracy: Observed Vs Simulated Data

One common reason for discrepancies between page speed testing tools is the way the data is collected. In a lab test the network is throttled to a fixed speed, typically to match a slower mobile data connection. The way this throttling is achieved can impact your measurements.

PageSpeed Insights uses an approach called simulated throttling. Measurements are collected on a fast network connection. After that, a simulation of a slow 4G connection is applied to estimate how the page might have loaded on a mobile device.

You can install the Site Speed Chrome extension to view the original observed metrics when running a test on PageSpeed Insights.

Screenshot of Lighthouse reported and observed metrics, September 2024

Simulated data can sometimes be unreliable, as the Lighthouse simulation doesn’t handle all real life edge cases that can happen when opening a website.

For example, in this test we can see that the Largest Contentful Paint metric is reported as one second worse than the values observed when opening the page in Chrome.

However, the original values for the First Continental Paint and for the Largest Contentful Paint metrics were identical. This suggests that the simulated metrics could potentially not match what real users experience.

You can check the settings section of the Lighthouse report to see if the metrics were measured as reported or if a simulation has been applied.

Screenshot of Lighthouse settings, September 2024

If you want to get reliable page speed data, the free DebugBear page speed test is built to provide the most accurate insight. Collecting real measurements takes a bit longer than running a simulation, but it will also help you make the best decisions when optimizing your website speed.

Why Des The Real User Data Not Match The Lighthouse Test Rresults?

When testing your website on PageSpeed Insights you’ll often find that the real user metrics are much better than those reported by the synthetic Lighthouse test. Why is that?

That’s because the Lighthouse test uses a very slow network connection. The CrUX Core Web Vitals data looks at the slowest 25% of user experiences on your website, but typically, even those visits come from a device that has a decent network connection.

So, a bad Lighthouse performance score doesn’t necessarily mean that you’ll fail Google’s Core Web Vitals assessment. But it can indicate that some users are having a poor experience and that there’s more room for improvement.

Screenshot of real user and lab-based performance metrics in PageSpeed Insights, September 2024

5. Use The PSI API To Automate Performance Testing

Got a lot of pages on your website you want to test? You can use the PageSpeed Insights API to automatically run website tests in bulk.

The API provides more detailed performance metrics and details on each Lighthouse audit. For example, you can use the API to see the most common performance recommendations across your website.

There’s even a way to access PageSpeed Insights data directly in Google Sheets.

JSON page speed data reported by the PageSpeed Insights API, September 2024

6. Know When To Reach For A Different Tool

PageSpeed Insights is a great tool to run a quick performance for a specific URL on your website. However, as we’ve seen above, this data comes with some limitations.

If you just want to get a site-wide overview of Core Web Vitals on your website, the quickest way to find this data is using Google Search Console.

Search Console will show you exactly how many pages on your website are slow or need to be improved.

Screenshot of Core Web Vitals data in Google Search Console, September 2024

Need to dive deep into CPU performance, for example to optimize the new Interaction to Next Paint metric?

The Performance tab in Chrome’s developer tools provides a detailed analysis of all kinds of CPU processing that happens on your website.

Screenshot of a website performance profile in Chrome DevTools, September 2024

Finally, if you want to optimize how different resources are loaded on your website, the DebugBear website speed test can be invaluable.

This test can provide a detailed report on what resources are loaded by your website, when they load, and how they impact rendering.

Screenshot of a website request waterfall in DebugBear, September 2024

How To Always Stay Ahead Of Your Website Speed

PageSpeed Insights and other performance tests are a great starting point for optimizing your website. However, without continuous monitoring, you risk reintroducing problems without noticing.

DebugBear is a monitoring platform for Core Web Vitals that lets you continuously test both your own website and those of your competitors.

Screenshot of the DebugBear performance dashboard, September 2024

In addition to scheduled lab testing, DebugBear also keeps track of Google CrUX data and collects real user analytics directly on your website.

The real user data provides a wide range of insight to not just help you keep track of performance but actively improve it:

  • See what LCP subpart is causing the biggest delay for your visitors
  • Find specific interactions and scripts that cause a poor Interaction to Next Paint score
  • Identify specific countries or devices where performance is worse than usual
Screenshot of real user monitoring data in DebugBear, September 2024

Deliver A Great User Experience

PageSpeed Insights is a helpful tool for any website owner, not just telling you how fast your website is in the real world, but also giving you concrete advice on how to optimize it.

However, if you’d like to go beyond the data PSI provides and test your website continuously, you can sign up for a free 14-day DebugBear trial.

This article has been sponsored by DebugBear, and the views presented herein represent the sponsor’s perspective.

Ready to start optimizing your website? Sign up for DebugBear and get the data you need to deliver great user experiences.


Image Credits

Featured Image: Image by DebugBear. Used with permission.

Leverage Search Intent & Boost Your Visibility With These Expert SEO Strategies via @sejournal, @lorenbaker

Struggling to rank for your target keywords? You’re not alone.

The SEO landscape is more complex than ever, with search intent evolving and SERP features constantly changing.

So, how do you make sure your content aligns with Google’s evolving expectations?

Check out our webinar on September 25, 2024: “Navigating SERP Complexity: How to Leverage Search Intent for SEO.”

Tom Capper of STAT will discuss the role of search intent in SEO and how to use it to climb in the right SERPs for your brand.

Why This Webinar Is A Must-Attend Event

Ranking isn’t just about keywords anymore—it’s about understanding the intent behind each search.

We’ll cover:

  1. How intent is nuanced, and many keywords can support multiple intents.
  2. Why the same keyword can have a different intent depending where it was searched from, and on what device.
  3. The differences in SERP features depending on intent, and how this impacts your content strategy.

Expert Insights From Tom Capper

Leading this session is Tom Capper from STAT Search Analytics. 

Capper will dive deep into searcher motivations using first-party research data and provide actionable insights to help you improve your site’s organic visibility.

Reserve your spot and find out more about how these insights can impact your ranking.

Who Should Attend?

This webinar is perfect for:

  • SEO professionals looking to take their strategies to the next level
  • Content managers and strategists wanting to increase the effectiveness of their work
  • Enterprise professionals and digital marketers looking to blend branding, marketing, and SEO for a unified customer experience
  • Anyone interested in search results and consumer behavior

Live Q&A: Get Your Questions Answered

Following the presentation, Tom will host a live Q&A session. 

This is your chance to clarify misconceptions surrounding the intersection of content, search intent, and the SERPs and get expert advice on optimizing your strategies.

Don’t Miss Out!

Understanding search intent is critical to staying competitive in SEO. Reserve your spot today to ensure you’re not left behind.

Can’t attend live? Sign up anyway for the recording.

Get ready to unlock new SEO opportunities and boost your rankings. See you there!

Google On Why Simple Factors Aren’t Ranking Signals via @sejournal, @martinibuster

Google’s John Mueller affirmed in a LinkedIn post that two site characteristics that could be perceived as indicative of site quality aren’t ranking factors, suggesting that other perceived indicators of quality may not be either.

Site Characteristics And Ranking Factors

John Mueller posted something interesting on LinkedIn because it offers insight into how an attribute of quality sometimes isn’t enough to be an actual ranking factor. His post also encourages a more realistic consideration of what should be considered a signal of quality and what is simply a characteristic of a site.

The two characteristics of site quality that Mueller discussed are valid HTML and typos (typographical errors, commonly in reference to spelling errors). His post was inspired by an analysis of 200 home pages of the most popular websites that found that only 0.5% of which had valid HTML. That means that out of the 200 of the most popular sites, only 1 home page was written with valid HTML.

John Mueller said that a ranking factor like valid HTML would be a low bar, presumably because spammers can easily create web page templates that use valid HTML. Mueller also made the same observation about typos.

Valid HTML

Valid HTML means that the code underlying a web page follows all of the rules for how HTML should be used. What constitutes valid HTML is defined by the W3C (World Wide Web Consortium), the international standards making body for the web. HTML, CSS, and Web Accessibility are examples of standards that the W3C creates. The validity of HTML can be tested at the W3C Markup Validation Service which is available at validator.w3.org.

Is Valid HTML A Ranking Factor?

The post begins by stating that a commonly asked question is whether valid HTML is a ranking factor or some other kind of factor for Google Search. It’s a valid question because valid HTML could be seen as a characteristic of quality.

He wrote:

“Every now and then, we get questions about whether “valid HTML” is a ranking factor, or a requirement for Google Search.

Jens has done regular analysis of the validity of the top websites’ homepages, and the results are sobering.”

The phrase, “the results are sobering” means that the results that most home pages use invalid HTML is surprising and possibly cause for consideration.

Given how virtually all content management systems do not generate valid HTML, I’m somewhat surprised that even one site out of 200 used valid HTML. I would expect a number closer to zero.

Mueller goes on to note that valid HTML is a low bar for a ranking factor:

“…this is imo a pretty low bar. It’s a bit like saying professional writers produce content free of typos – that seems reasonable, right? Google also doesn’t use typos as a ranking factor, but imagine you ship multiple typos on your homepage? Eww.

And, it’s trivial to validate the HTML that a site produces. It’s trivial to monitor the validity of important pages – like your homepage.”

Ease Of Achieving Characteristic Of Quality

There have been many false signals of quality promoted and abandoned by SEOs, the most recent one being “authorship” and “content reviews” that are supposed to show that an authoritative author wrote an article and that the article was checked by someone who is authoritative. People did things like invent authors with AI generated images that are associated to fake LinkedIn profiles in the naïve belief that adding an author to the article will trick Google into awarding ranking factor points (or whatever, lol).

The authorship signal turned out to be a misinterpretation of Google’s Search Quality Raters Guidelines and a big waste of a lot of people’s time. If SEOs had considered how easy it was to create an “authorship” signal it would have been apparent to more people that it was a trivial thing to fake.

So, one takeaway from Mueller’s post can be said to be that if there’s a question about whether something is a ranking factor, first check if Google explicitly says it’s a ranking factor and if not then consider if literally any spammer can achieve that “something” that an SEO claims is a ranking factor. If it’s a trivial thing to achieve then there’s a high likelihood it’s not a ranking factor.

There Is Still Value To Be Had From Non-Ranking Factors

The fact that something is relatively easy to fake doesn’t mean that web publishes and site owners should stop doing it. If something is good for users and helps to build trust then it’s likely a good idea to keep doing it. Just because something is not a ranking factor doesn’t invalidate the practice.  It’s always a good practice in the long run to keep doing activities that build trust in the business or the content, regardless of whether it’s a ranking factor or not.  Google tries to pick up on the signals that users or other websites give in order to determine if a website is high quality, useful, and helpful, so anything that generates trust and satisfaction is likely a good thing.

Read John Mueller’s post on LinkedIn here.

Featured Image by Shutterstock/stockfour

Why Content Is Important For SEO via @sejournal, @lorenbaker

Content is SEO. More specifically, it’s one side of the SEO relationship. One core function of search engines is to connect users with the information they’re looking for. That information might be a product listing, a review, a news story, an image, or a video.

The other core function of search engines is to retain users.

Search engines retain users by ensuring their confidence and trust in the displayed results. Over time, they build expectations that using their platform is a safe, streamlined experience that quickly leads users to what they want.

SEO success depends on being found by your target audience for what they are looking for and consistently providing a satisfying user experience based on the context of the queries they type into search engines.

Search Is Built On Content

The core function of search engines is to help users find information. Search engines first discover webpages, they parse and render and they then add them to an index. When a user inputs a query, search engines retrieve relevant webpages in the index and then “rank” them.

Search engines need to know what pages are about and what they contain in order to serve them to the right users. In concept, they do this quite simply: They examine the content. The real process behind this is complicated, executed by automated algorithms and evaluated with human feedback.

Google constantly adjusts and updates it algorithms with the goal of ensuring the most relevant content is served to searchers.

This relationship between searchers, search engines, and websites, has come to define the internet experience for most users. Unless you know the exact URL of the website you intend to visit, you need must find it via a third party. That could be social media, a search engine, or even discovering the website offline and then typing it in. This is called a “referral,” and Google sends 64% of all website referrals in the U.S. Microsoft and Bing send the next largest amount of referrals, followed by YouTube.

Getting discovered by people who don’t already know you depends on search engines, and search engines depend on content.

The SEO Value Of Content

Google has said it prioritizes user satisfaction.

It’s confirmed that user behavior signals impact ranking.

At this point, whether this relationship is causal or correlative doesn’t matter. You must prioritize user experience and satisfaction because it’s a key indicator of SEO success.

Written language is still the primary way users interact with search engines and how algorithms understand websites. Google algorithms can interpret audio and videos, but written text is core to SEO functionality.

Enticing clicks and engaging users through content that satisfies their queries is the baseline of SEO. If your pages can’t do that, you won’t have success.

High-quality content and user experiences aren’t just important for SEO; they’re prerequisites.

This is true for all advertising and branding. Entire industries and careers are built on the skills to refine the right messaging and put it in front of the right people.

Evidence For The SEO Value Of Content

Google highlights the importance of content in its “SEO fundamentals” documentation. It advises that Google’s algorithms look for “helpful, reliable information that’s primarily created to benefit people,” and provides details about how to self-assess high-quality content.

  • Content, and how well it matches a user’s needs, is one of the core positive and negative factors in Google’s ranking systems. It updates systems to reduce content it deems to be unhelpful and prioritize content it deems to be helpful.
  • In fact, Google’s analysis of the content may determine whether a page enters the index at all to become eligible to rank. If you work hard to provide a good experience and serve the needs of your users, search engines have more reason to surface your content and may do so more often.
  • A 2024 study in partnership between WLDM, ClickStream, and SurferSEO suggests that the quality of your coverage on a topic is highly correlated with rankings.

Content And User Behavior

Recent developments in the SEO industry, such as the Google leak, continue to highlight the value of both content and user experience.

Google values user satisfaction to determine the effectiveness and quality of webpages and does seem to use behavioral analysis in ranking websites. It also focuses on the user intent of queries and whether a specific intent is served by a particular resource.

The satisfaction of your users is, if not directly responsible for SEO performance, highly correlated with it.

Many factors affect user experience and satisfaction. Website loading speed and other performance metrics are part of it. Intrusive elements of the page on the experience are another.

Content, however, is one of the primary determiners of a “good” or “bad” experience.

  • Does the user find what they’re looking for? How long does it take?
  • Is the content accurate and complete?
  • Is the content trustworthy and authoritative?

The answers to these questions reflect whether the user has a good or bad experience with your content, and this determines their behavior. Bad experiences tend to result in the user leaving without engaging with your website, while good experiences tend to result in the user spending more time on the page or taking action.

This makes content critical not only to your SEO efforts on search engines but also to your website’s performance metrics. Serving the right content to the right users in the right way impacts whether they become leads, convert, or come back later.

Leaning into quality and experience is a win all around. Good experiences lead to desirable behaviors. These behaviors are strong indications of the quality of your website and content. They lead to positive outcomes for your business and are correlated with successful SEO.

What Kinds Of Content Do You Need?

Successful content looks different for each goal you have and the different specific queries you’re targeting.

Text is still the basis of online content when it comes to search. Videos are massively popular. YouTube is the second-most popular search engine in the world. However, in terms of referrals, it only sends 3.5% of referral traffic to the web in the U.S. In addition, videos have titles, and these days, most have automated transcripts. These text elements are critical for discovery.

That isn’t to say videos and images aren’t popular. Video, especially “shorts” style videos, is an increasingly popular medium. Cisco reported that video made up 82% of all internet traffic in 2022. So you absolutely shoulder consider images and video as part of your content strategy to best serve your audiences and customers.

Both can enhance text-based webpages and stand on their own on social platforms.

But for SEO, it’s critical to remember that Google search sends the most referral traffic to other websites. Text content is still the core of a good SEO strategy. Multi-modal AI algorithms are getting very good at translating information between various forms of media, but text content remains critical for several reasons:

  • Plain text has high accessibility. Screen readers can access it, and it can be resized easily.
  • Text is the easiest way for both people and algorithms to analyze semantic connections between ideas and entities.
  • Text doesn’t depend on device performance like videos and images might.
  • Text hyperlinks are very powerful SEO tools because they convey direct meaning along with the link.
  • It’s easier to skim through text than video.

Text content is still dominant for SEO. But you should not ignore other content. Images, for example, make for strong link building assets because they’re attractive and easily sharable. Accompanying text with images and video accommodates a variety of user preferences and can help capture attention when plain text might not.

Like everything else, it’s down to what best serves users in any given situation.

SEO Content: Serving Users Since Search Was A Thing

Search engines match content to the needs of users.

Content is one-third of this relationship: user – search engine – information.

You need content to perform SEO, and any digital marketing activity successfully.

The difficulty comes from serving that perfect content for the perfect situation.

So read “How To Create High-Quality Content” next.

Read More:


Featured Image: Roman Samborskyi/Shutterstock

Why & How To Track Google Algorithm Updates via @sejournal, @lorenbaker

Google constantly evaluates and updates its algorithms. There can be hundreds or even thousands of individual changes per year.

Google does confirm some of the major updates, such as site reputation abuse, the March 2024 core update, and the November 2023 reviews update.

But, often, Google will not officially confirm an update, and these are only picked up through high volatility in the SERPs.

For example, in May of 2024, Lily Ray observed huge changes in traffic to a dozen publisher sites using rank tracking tools. Google rejected the idea of an algorithm update.

Google rejects the idea of an algorithm update observed by SEO tools.Google rejects the idea of an algorithm update observed by SEO tools.

The volatility mentioned in the tweet was observed around May 7, a day after Google announced that it rolled out a reputation abuse update with manual actions, with the algorithmic part following later.

Since it didn’t mention specific dates, many assumed that those websites were hit by the reputation abuse algorithmic rollout. However, SearchLiaison responded and refuted that assumption, leaving many SEO pros in a state of confusion.

A lot of common SEO advice you’ll see (especially from Google) amounts to “don’t chase algorithms, just do what’s best for the user” – but algorithms can have a catastrophic impact on SEO performance (sometimes unjustly).

For this reason, if you are managing a site for a brand, you need to act quickly if there is an update.

Knowing when an update hits and understanding each update will help you to adjust your strategy as needed, to avoid being impacted in future updates and also to try and recover quickly if you do have a negative impact.

Why You Should Track Google Updates

Understanding algorithms and updates is a core SEO skill. Occasionally, Google releases an update that is consequential enough to get a name (e.g., FloridaPanda, Penguin, RankBrain), and significantly impact how Google search works. You don’t want to get caught out by a big update, which means you should analyze the history of the algorithms to understand their future trajectory.

An algorithm change or update primarily impacts your website’s organic visibility in Google Search. Mostly, that comes in the form of rankings. But updates can cause disruption in other ways, too, for example by adding Search features to a particular search engine results page (SERP) that reduce click-through rates and traffic.

Tracking and understanding Google updates helps you adjust for sudden performance instability. It also helps you create SEO strategies that will be effective in the long term. Understanding where the algorithms have been helps you project where they might go. This will help you avoid risky SEO practices and reduce the risk of an update significantly impacting your website.

Recovering from updates that impact you negatively takes work and time. If you track updates, you can understand why your site’s ranking might have changed and take the necessary steps to recover as quickly as possible.

Resources For Tracking Google Algorithm Updates

Here are resources that can make your life easier and help you keep track of Google algorithm updates.

Google Search Status Dashboard

Google search dashboardImage from Google search dashboard, June 2024

The advantage of this dashboard is that it also tracks indexing and crawling incidents alongside algorithm updates in the Ranking section.

You can subscribe to updates using this RSS feed it provides.

Keep an eye on this resource to stay updated on the latest changes and incidents straight from Google.

X (Formerly Twitter)

Ten years ago, Matt Cutts was the best person at Google to follow as he regularly kept the SEO community informed about changes to search.

This role is now performed by Google SearchLiaison, which is managed by Danny Sullivan.

Google SearchLiaison's Page on XGoogle SearchLiaison’s Page on X

Make sure you follow the real @searchliaison page that has a verified badge on the profile.

If you have questions regarding Google’s algorithm updates, you may post your question on X by tagging @searchliaison, and you may be lucky to get answers directly from Google. Try to be comprehensive and provide as many details about your issue as possible to increase your chances of getting a response.

Other than the official page, you may want to follow Barry Schwartz (@rustybrick) and Marie Haynes (@Marie_Haynes), who are always on the lookout for news about algorithm changes.

Search Engine Journal

History of Google Algorithm UpdatesHistory of Google Algorithm Updates

Search Engine Journal has a dedicated page about the history of Google’s algorithm updates – from 2003 to the present. It includes the following information:

  • Algorithm name.
  • The rollout date.
  • A brief overview of the impact.
  • Whether it is confirmed or unconfirmed.
  • Related publications and official announcements so you can dive deeper and understand the changes.

You can also sign up for SEJ’s newsletters, and we’ll keep you posted on every major algorithm update.

Google isn’t a fan of third-party tools that track algorithm updates. It warns the SEO community that they are prone to errors and may have false positive detections.

I can see why Google disliked them, as they are crawling Google SERPs regularly to gather data – which, of course, Google doesn’t like. 😀

It is often true that they report volatile changes in search result pages. Still, in most cases, these tools report accurately by providing “volatility scores,” representing how much the SERP has changed.

Below is a table detailing the SERP volatility levels for various tools:

Now, let’s review a few tools you can use to track Google’s algorithm updates.

1. MozCast

Screenshot of MozCast from moz.comScreenshot of MozCast from moz.com

MozCast makes rank tracking fun in the style of a weather report.

It compares the rankings of the same set of keywords on two consecutive days and calculates how much the positions of these keywords have moved up or down, translating into a temperature scale. Per their specifications, 70°F represents a normal, stable day, and higher temperatures indicate more drastic changes.

To get an idea of what temperature is considered high, I want to note that during the March core update, MozCast’s temperature was 108°F-115°F. On May 7, its score was 90°F, then went up to 111°F, indicating that MozCast could detect movements.

So, when MozCast temperature is close to 100°F, it is quite high.

2. Semrush Sensor

Screenshot from Semrush.comScreenshot from Semrush.com

Semrush Sensor is a powerful tool designed to help you understand and track fluctuations in rankings.

Similar to MozCast, it monitors a fixed set of keywords and how much the search results for these keywords change by the end of each day. But it provides richer information by industries and locations.

Another highly valuable feature of the Semrush sensor is the report of winners and losers, which can help you run a quick competitive analysis to see websites benefiting or suffering from recent changes.

Its scale varies from 0 to 10. Usually, during core algorithm updates, the score is between 8 and 10. On May 7, its score was around 9.5 out of 10, which means there was an earthquake in SERPs.

3. Similarweb

Screenshot from Similarweb.comScreenshot from Similarweb.com

Similarweb monitors more than 10,000 domains and keywords on a daily basis to identify ranking patterns and track volatility in Google’s desktop and mobile search results.

Here is how to read its graphs:

  • The numbers on the graph indicate the level of ranking fluctuations on specific dates.
  • A higher number means more significant changes in rankings.
  • Orange signals a moderate risk.
  • Red indicates a high risk.

Again, to give you an idea of what risk level is considered high, I want to mention that during Google’s March core update, the risk level metric was 65. On May 11, the risk level metric was 71, which is high. We can conclude that Similarweb was able to detect the anomaly observed by the SEO community.

4. Accuranker ‘Grump’ Rating

Screenshot from accuranker.comScreenshot from accuranker.com

Accuranker is another great tool for observing Google SERP volatilities.

They have a fun scoring scale:

  • Grumpy (0-10): Google is chilled.
  • Cautious (10-12): Normal activity.
  • Grumpy (12-15): More than usual.
  • Furious (15+): High fluctuations in SERP.

One advantage over others is that they let you go back as far as you want, providing historical data back to 2016, and the data is updated in real-time.

In contrast to several other sensors, it provides details on how Accuranker calculates its rating:

  • It monitors a set of 30,000 randomly selected keywords.
  • It splits the keyword selection set between mobile and desktop searches (15,000 each).
  • For each keyword, it analyzes the top 100 search results.
  • The final index number for the keyword is the total sum of the position differences for each keyword divided by the number of results (typically 100).

A higher index number means more significant fluctuations in the rankings. For example, during the core update, it is in the order of ~14, which is more than usual. On May 7-9, the tool scored “Google is chilled” and ‘Cautious’ with a score of ~9.

5. Advanced Web Rankings Google Algorithm Changes

Screenshot from advancedwebranking.comScreenshot from advancedwebranking.com

Advanced Web Rankings monitors the ranking changes of approximately 400,000 desktop keywords and 200,000 mobile keywords across various countries.

You can segment the data countries, devices, and industries, and look up historical data by going back as much as you want by selecting a custom date period.

The tool calculates the Volatility (KPI), which has the following areas:

  • Low Volatility: Indicates insignificant changes.
  • Medium Volatility: Represents moderate changes in SERPs, which could be due to minor algorithm updates or other factors.
  • High Volatility: This means high fluctuations in SERP often correlated with major Google algorithm updates.

During the March core update, it detected high volatility with a score of 7.3 and medium volatility with a score of 4-5 on May 7-9.

6. CognitiveSEO Signals

Screenshot from cognitiveseo.comScreenshot from cognitiveseo.com

CognitiveSEO Signals monitors over 170,000 keywords. These keywords are randomly selected to track ranking fluctuations in desktop, mobile, and local search results.

Again, it doesn’t disclose how it calculates the volatility score, but it has a nice chart showing days with high fluctuations in red.

During the recent March core update and on May 7-9, it detected high volatility, with scores of 70 and 75, respectively.

7. Algoroo

Screenshot from algoroo.comScreenshot from algoroo.com

Algoroo is another tool to track Google’s algorithm updates, which is built and maintained by Dejan.

It doesn’t disclose how tracking works. What we know is that it tracks selected keywords and calculates their ranking movements.

Reading data is really simple; when bars are in red, it means high fluctuations.

During the recent March core update, medium volatility was detected, and nothing unusual but normal activity on May 7-9.

What To Do After An Algorithm Update

There are six things you should always remember when algorithm updates (whether confirmed or unconfirmed) negatively impact your website:

  • Don’t jump and perform sitewide changes in panic mode.
  • Check the website’s technical setup to ensure that your traffic didn’t drop due to the server being down or your developer accidentally blocking it via robots.txt or noindexing mistakenly.
  • Be patient and collect data.
  • Observe how your competitors are affected by the update to find any patterns.
  • Read credible sources (like Search Engine Journal) to gain insights and see what the SEO experts have to say.
  • Make adjustments to your SEO strategy and tactics as necessary.

It’s also important to remember that Google’s algorithms are constantly changing.

What impacts your rankings today could change in a few days, a week, or a month.

For more in-depth information, check out our guides:


Featured Image: salarko/Shutterstock

The Expert SEO Guide To URL Parameter Handling via @sejournal, @jes_scholz

In the world of SEO, URL parameters pose a significant problem.

While developers and data analysts may appreciate their utility, these query strings are an SEO headache.

Countless parameter combinations can split a single user intent across thousands of URL variations. This can cause complications for crawling, indexing, visibility and, ultimately, lead to lower traffic.

The issue is we can’t simply wish them away, which means it’s crucial to master how to manage URL parameters in an SEO-friendly way.

To do so, we will explore:

What Are URL Parameters?

url parameter elementsImage created by author

URL parameters, also known as query strings or URI variables, are the portion of a URL that follows the ‘?’ symbol. They are comprised of a key and a value pair, separated by an ‘=’ sign. Multiple parameters can be added to a single page when separated by an ‘&’.

The most common use cases for parameters are:

  • Tracking – For example ?utm_medium=social, ?sessionid=123 or ?affiliateid=abc
  • Reordering – For example ?sort=lowest-price, ?order=highest-rated or ?so=latest
  • Filtering – For example ?type=widget, colour=purple or ?price-range=20-50
  • Identifying – For example ?product=small-purple-widget, categoryid=124 or itemid=24AU
  • Paginating – For example, ?page=2, ?p=2 or viewItems=10-30
  • Searching – For example, ?query=users-query, ?q=users-query or ?search=drop-down-option
  • Translating – For example, ?lang=fr or ?language=de

SEO Issues With URL Parameters

1. Parameters Create Duplicate Content

Often, URL parameters make no significant change to the content of a page.

A re-ordered version of the page is often not so different from the original. A page URL with tracking tags or a session ID is identical to the original.

For example, the following URLs would all return a collection of widgets.

  • Static URL: https://www.example.com/widgets
  • Tracking parameter: https://www.example.com/widgets?sessionID=32764
  • Reordering parameter: https://www.example.com/widgets?sort=latest
  • Identifying parameter: https://www.example.com?category=widgets
  • Searching parameter: https://www.example.com/products?search=widget

That’s quite a few URLs for what is effectively the same content – now imagine this over every category on your site. It can really add up.

The challenge is that search engines treat every parameter-based URL as a new page. So, they see multiple variations of the same page, all serving duplicate content and all targeting the same search intent or semantic topic.

While such duplication is unlikely to cause a website to be completely filtered out of the search results, it does lead to keyword cannibalization and could downgrade Google’s view of your overall site quality, as these additional URLs add no real value.

2. Parameters Reduce Crawl Efficacy

Crawling redundant parameter pages distracts Googlebot, reducing your site’s ability to index SEO-relevant pages and increasing server load.

Google sums up this point perfectly.

“Overly complex URLs, especially those containing multiple parameters, can cause a problems for crawlers by creating unnecessarily high numbers of URLs that point to identical or similar content on your site.

As a result, Googlebot may consume much more bandwidth than necessary, or may be unable to completely index all the content on your site.”

3. Parameters Split Page Ranking Signals

If you have multiple permutations of the same page content, links and social shares may be coming in on various versions.

This dilutes your ranking signals. When you confuse a crawler, it becomes unsure which of the competing pages to index for the search query.

4. Parameters Make URLs Less Clickable

parameter based url clickabilityImage created by author

Let’s face it: parameter URLs are unsightly. They’re hard to read. They don’t seem as trustworthy. As such, they are slightly less likely to be clicked.

This may impact page performance. Not only because CTR influences rankings, but also because it’s less clickable in AI chatbots, social media, in emails, when copy-pasted into forums, or anywhere else the full URL may be displayed.

While this may only have a fractional impact on a single page’s amplification, every tweet, like, share, email, link, and mention matters for the domain.

Poor URL readability could contribute to a decrease in brand engagement.

Assess The Extent Of Your Parameter Problem

It’s important to know every parameter used on your website. But chances are your developers don’t keep an up-to-date list.

So how do you find all the parameters that need handling? Or understand how search engines crawl and index such pages? Know the value they bring to users?

Follow these five steps:

  • Run a crawler: With a tool like Screaming Frog, you can search for “?” in the URL.
  • Review your log files: See if Googlebot is crawling parameter-based URLs.
  • Look in the Google Search Console page indexing report: In the samples of index and relevant non-indexed exclusions, search for ‘?’ in the URL.
  • Search with site: inurl: advanced operators: Know how Google is indexing the parameters you found by putting the key in a site:example.com inurl:key combination query.
  • Look in Google Analytics all pages report: Search for “?” to see how each of the parameters you found are used by users. Be sure to check that URL query parameters have not been excluded in the view setting.

Armed with this data, you can now decide how to best handle each of your website’s parameters.

SEO Solutions To Tame URL Parameters

You have six tools in your SEO arsenal to deal with URL parameters on a strategic level.

Limit Parameter-based URLs

A simple review of how and why parameters are generated can provide an SEO quick win.

You will often find ways to reduce the number of parameter URLs and thus minimize the negative SEO impact. There are four common issues to begin your review.

1. Eliminate Unnecessary Parameters

remove unnecessary parametersImage created by author

Ask your developer for a list of every website’s parameters and their functions. Chances are, you will discover parameters that no longer perform a valuable function.

For example, users can be better identified by cookies than sessionIDs. Yet the sessionID parameter may still exist on your website as it was used historically.

Or you may discover that a filter in your faceted navigation is rarely applied by your users.

Any parameters caused by technical debt should be eliminated immediately.

2. Prevent Empty Values

no empty parameter valuesImage created by author

URL parameters should be added to a URL only when they have a function. Don’t permit parameter keys to be added if the value is blank.

In the above example, key2 and key3 add no value, both literally and figuratively.

3. Use Keys Only Once

single key usageImage created by author

Avoid applying multiple parameters with the same parameter name and a different value.

For multi-select options, it is better to combine the values after a single key.

4. Order URL Parameters

order url parametersImage created by author

If the same URL parameter is rearranged, the pages are interpreted by search engines as equal.

As such, parameter order doesn’t matter from a duplicate content perspective. But each of those combinations burns crawl budget and split ranking signals.

Avoid these issues by asking your developer to write a script to always place parameters in a consistent order, regardless of how the user selected them.

In my opinion, you should start with any translating parameters, followed by identifying, then pagination, then layering on filtering and reordering or search parameters, and finally tracking.

Pros:

  • Ensures more efficient crawling.
  • Reduces duplicate content issues.
  • Consolidates ranking signals to fewer pages.
  • Suitable for all parameter types.

Cons:

  • Moderate technical implementation time.

Rel=”Canonical” Link Attribute

rel=canonical for parameter handlingImage created by author

The rel=”canonical” link attribute calls out that a page has identical or similar content to another. This encourages search engines to consolidate the ranking signals to the URL specified as canonical.

You can rel=canonical your parameter-based URLs to your SEO-friendly URL for tracking, identifying, or reordering parameters.

But this tactic is not suitable when the parameter page content is not close enough to the canonical, such as pagination, searching, translating, or some filtering parameters.

Pros:

  • Relatively easy technical implementation.
  • Very likely to safeguard against duplicate content issues.
  • Consolidates ranking signals to the canonical URL.

Cons:

  • Wastes crawling on parameter pages.
  • Not suitable for all parameter types.
  • Interpreted by search engines as a strong hint, not a directive.

Meta Robots Noindex Tag

meta robots noidex tag for parameter handlingImage created by author

Set a noindex directive for any parameter-based page that doesn’t add SEO value. This tag will prevent search engines from indexing the page.

URLs with a “noindex” tag are also likely to be crawled less frequently and if it’s present for a long time will eventually lead Google to nofollow the page’s links.

Pros:

  • Relatively easy technical implementation.
  • Very likely to safeguard against duplicate content issues.
  • Suitable for all parameter types you do not wish to be indexed.
  • Removes existing parameter-based URLs from the index.

Cons:

  • Won’t prevent search engines from crawling URLs, but will encourage them to do so less frequently.
  • Doesn’t consolidate ranking signals.
  • Interpreted by search engines as a strong hint, not a directive.

Robots.txt Disallow

robots txt disallow for parameter handlingImage created by author

The robots.txt file is what search engines look at first before crawling your site. If they see something is disallowed, they won’t even go there.

You can use this file to block crawler access to every parameter based URL (with Disallow: /*?*) or only to specific query strings you don’t want to be indexed.

Pros:

  • Simple technical implementation.
  • Allows more efficient crawling.
  • Avoids duplicate content issues.
  • Suitable for all parameter types you do not wish to be crawled.

Cons:

  • Doesn’t consolidate ranking signals.
  • Doesn’t remove existing URLs from the index.

Move From Dynamic To Static URLs

Many people think the optimal way to handle URL parameters is to simply avoid them in the first place.

After all, subfolders surpass parameters to help Google understand site structure and static, keyword-based URLs have always been a cornerstone of on-page SEO.

To achieve this, you can use server-side URL rewrites to convert parameters into subfolder URLs.

For example, the URL:

www.example.com/view-product?id=482794

Would become:

www.example.com/widgets/purple

This approach works well for descriptive keyword-based parameters, such as those that identify categories, products, or filters for search engine-relevant attributes. It is also effective for translated content.

But it becomes problematic for non-keyword-relevant elements of faceted navigation, such as an exact price. Having such a filter as a static, indexable URL offers no SEO value.

It’s also an issue for searching parameters, as every user-generated query would create a static page that vies for ranking against the canonical – or worse presents to crawlers low-quality content pages whenever a user has searched for an item you don’t offer.

It’s somewhat odd when applied to pagination (although not uncommon due to WordPress), which would give a URL such as

www.example.com/widgets/purple/page2

Very odd for reordering, which would give a URL such as

www.example.com/widgets/purple/lowest-price

And is often not a viable option for tracking. Google Analytics will not acknowledge a static version of the UTM parameter.

More to the point: Replacing dynamic parameters with static URLs for things like pagination, on-site search box results, or sorting does not address duplicate content, crawl budget, or internal link equity dilution.

Having all the combinations of filters from your faceted navigation as indexable URLs often results in thin content issues. Especially if you offer multi-select filters.

Many SEO pros argue it’s possible to provide the same user experience without impacting the URL. For example, by using POST rather than GET requests to modify the page content. Thus, preserving the user experience and avoiding SEO problems.

But stripping out parameters in this manner would remove the possibility for your audience to bookmark or share a link to that specific page – and is obviously not feasible for tracking parameters and not optimal for pagination.

The crux of the matter is that for many websites, completely avoiding parameters is simply not possible if you want to provide the ideal user experience. Nor would it be best practice SEO.

So we are left with this. For parameters that you don’t want to be indexed in search results (paginating, reordering, tracking, etc) implement them as query strings. For parameters that you do want to be indexed, use static URL paths.

Pros:

  • Shifts crawler focus from parameter-based to static URLs which have a higher likelihood to rank.

Cons:

  • Significant investment of development time for URL rewrites and 301 redirects.
  • Doesn’t prevent duplicate content issues.
  • Doesn’t consolidate ranking signals.
  • Not suitable for all parameter types.
  • May lead to thin content issues.
  • Doesn’t always provide a linkable or bookmarkable URL.

Best Practices For URL Parameter Handling For SEO

So which of these six SEO tactics should you implement?

The answer can’t be all of them.

Not only would that create unnecessary complexity, but often, the SEO solutions actively conflict with one another.

For example, if you implement robots.txt disallow, Google would not be able to see any meta noindex tags. You also shouldn’t combine a meta noindex tag with a rel=canonical link attribute.

Google’s John Mueller, Gary Ilyes, and Lizzi Sassman couldn’t even decide on an approach. In a Search Off The Record episode, they discussed the challenges that parameters present for crawling.

They even suggest bringing back a parameter handling tool in Google Search Console. Google, if you are reading this, please do bring it back!

What becomes clear is there isn’t one perfect solution. There are occasions when crawling efficiency is more important than consolidating authority signals.

Ultimately, what’s right for your website will depend on your priorities.

url parameter handling option pros and consImage created by author

Personally, I take the following plan of attack for SEO-friendly parameter handling:

  • Research user intents to understand what parameters should be search engine friendly, static URLs.
  • Implement effective pagination handling using a ?page= parameter.
  • For all remaining parameter-based URLs, block crawling with a robots.txt disallow and add a noindex tag as backup.
  • Double-check that no parameter-based URLs are being submitted in the XML sitemap.

No matter what parameter handling strategy you choose to implement, be sure to document the impact of your efforts on KPIs.

More resources: 


Featured Image: BestForBest/Shutterstock

International SEO Expansion: Best Practices Guide

Getting your international SEO strategy right can be an elusive feat.

There are a lot more factors at play than people give credit for, and it’s often a thankless job.

A successful international SEO strategy requires a deep knowledge of your company’s commercial strategy as well as technical SEO knowledge, cultural sensitivity, and excellent data skills.

Yet the industry often regards international SEO as just your hreflang setup.

In this article, I will distill the complexities of international SEO success into an actionable step-by-step list that will take you from beginner to advanced practitioner. Let’s begin!

Part I: Be Commercially Aware

1. Understand Why Your Company Is Going International

Companies can grow by expanding their products and services, focusing on gaining market penetration or expanding into new markets.

While your team’s goal might be traffic, leads, or revenue, the leadership team is likely working under a different set of parameters. Most of the time, leadership’s ultimate goal is to maximize shareholder value.

  • In founder-owned companies, growth goals might be slower and more sustainable, usually aimed at maintaining and growing profitability.
  • VC-owned companies have high growth goals because they must provide their investors with a return that’s higher than the stock market. This is what is known as the alpha, or your company’s ability to beat the market in growth.
  • Publicly traded companies are likely aiming to grow their share value.
  • Startups, depending on their maturity stage, are likely looking to prove product-market fit or expand their reach fast to show that their operations are scalable and have the potential to be profitable in the future. The goal of this is to aid in raising further capital from investors.

Understanding why businesses go international is essential for informing your SEO decisions. What’s best practice for SEO isn’t always what’s best for business.

You must adapt your strategy to your company’s growth model.

  • Companies choosing to grow sustainably and maintain profitability will likely expand more slowly to a market that resembles their core market.
  • VC-owned companies will be able to invest in a wider range of countries, with a smaller concern for providing their users with an experience on par with that of their core markets.
  • Startups can try to beat their competitors to market by expanding quickly and throwing a lot of money at the project, or they might be concerned with cash flow and try to expand fast but cut corners by using automatic translation.

2. Stack Rank Your Target Markets To Prioritize Your Investment

I promise I’ll get to hreflang implementation soon, but so much about international SEO has to do with commercial awareness – so bear with me; this will make you a better professional.

Many companies have different market tiers to reflect how much of a priority each market is. Market prioritization can happen using many different metrics, such as:

  • Average order value or lifetime customer value.
  • Amount of investment required.
  • Market size.
  • And market similarity.

American companies often prioritize developed English-speaking countries such as the UK, Canada, or Australia. These are most similar to their core market, and most of their market knowledge will be transferable.

After that, companies are likely to target large European economies, such as Germany and France. They might also target the LatAm market and Spain in the same effort.

The last prioritization tier can vary widely among companies, with a focus on the Nordic, Brazilian, or Asian markets.

Part II: Know Your Tech

3. Define Your International URL Structure

When doing international SEO, there are 4 different possible URL structures, each with its pros and cons.

ccTLD Structure

A ccTLD structure is set up to target different countries based on the domain type.

This structure is not ideal for companies that target different languages rather than different countries. For example, a .es website is targeting Spain, not the Spanish language.

An advantage to this kind of structure is that the ccTLD sends a very strong localization signal to search engines as to what market they are targeting, and they can lead to improved trust and CTR in your core country.

On the other hand, ccTLDs can dilute your site’s authority, as links will be spread across domains rather than concentrated on the .com.

gTLD With Subdirectories

This is my personal favorite when it comes to international SEO.

These URL structures can look like website.com/en if they’re targeting languages or website.com/en-gb if they’re targeting countries.

This configuration aggregates the authority you gain across your different territories into a single domain, it’s cheaper to maintain, and the .com TLD is widely recognizable by users worldwide.

On the other hand, this setup can look less personalized to people outside the US, who might wonder if you can service their markets.

gTLD With Subdomains

This setup involves placing international content on a subdomain like us.website.com. While once popular, it’s slipping in favor because it doesn’t bring anything unique to the table anymore.

This setup offers a clear signal to users and search engines about the intended audience of a specific subdomain.

However, subdomains often face issues with SEO, as Google tends to view them as separate entities. This separation can dilute link, similar to the ccTLD approach but without the geo-targeting advantages.

gTLD With Parameters

This is the setup where you add parameters at the end of the URL to indicate the language of the page, such as website.com/?lang=en.

I strongly advise against this setup, as it can present multiple technical SEO challenges and trust issues.

4. Understand Your Hreflang Setup

In the words of John Mueller: hreflang can be one of the most complex aspects of SEO.

Tweet by John Mueller talking about how hreflang can be one of the more complex aspects of SEO.Screenshot from Twitter, May 2024

Hreflang reminds me of a multilingual form of a canonical tag, where we tell search engines that one document is a version of the other and explain the relationship between them.

I find hreflang implementation very interesting from a technical point of view. Because development teams mostly manage it, and it can be very much hit or miss.

Often, hreflang is constructed from existing fields in your content management system (CMS) or content database.

You might find that your development team is pulling the HTML lang tag, which follows a different ISO standard than hreflang, leading to a broken implementation.

Other times, there is a field in your CMS that your development team pulls from to build your hreflang setup.

Finding out how your hreflang tags are generated can be extremely helpful in identifying the sources of different issues or mitigating potential risks.

So speak to your engineering team and ask them how you’re currently generating hreflang.

5. Implement Hreflang Without Errors

There are three ways to implement hreflang on your site:

  • On your sitemap.
  • Through your HTTP header.
  • On your HTML head.

The method most of us are most familiar with is the HTML head. And while you can use more than one method, they should match each other perfectly. Otherwise, you risk confusing search engines.

Here are some basic rules for getting it done correctly:

  • In your hreflang implementation, the URL must include domain and protocol.
  • You must follow the ISO 639-1 language codes – don’t go around making up your own.
  • Hreflang tags must be reciprocal. If the page you’re listing as a language alternative does not list you back, your implementation won’t work.
  • Audit your hreflang regularly. My favorite tool for this, since it added the hreflang cluster analysis and link graphs, is Ahrefs. For the record, Ahrefs is not paying me to say this; it’s a genuine recommendation and has helped me a lot in my work.
  • You should only have one page per language.
  • Your hreflang URLs should be self-canonicalizing and respond with a 200 code.

Follow the above rules, and you’ll avoid the most common hreflang mistakes that SEO pros make.

And if you’re interested in the technical SEO aspect beyond hreflang, I recommend reading Mind your language by Rob Owen.

Part III: Invest In Content Incrementally

6. Translate Your Top-performing Content Topics

Now that you have the basic commercial and technical knowledge covered, you’re ready to start creating a content strategy.

You likely have a wealth of content in your core market that can be recycled. But you want to focus on translating high-converting topics, not just any topic; otherwise, you might be wasting your budget!

Let’s go step by step.

Cluster Your Website’s Content By Topic

  • Crawl your site using your favorite SEO tool and extract the URL and H1.
  • Use ChatGPT to classify that list of URLs into topics. You might already know what you usually write about, so include those topics in your prompt. You don’t want to have a classification that’s too granular, so you can prompt chatGPT to only create groups with a minimum of 10 URLs (adjust this to reflect the size of your website) and class everything else as other. This is an example of what your prompt might look like: “I will provide you with a list of article titles and their corresponding URL. Classify this list into the following topics: survey best practices, research and analysis, employee surveys, market research and others. Return this in a table format with the URL, title and group name.”
  • Start a spreadsheet with all your URLs in the first column, titles in the second column, and the group they belong to in the third column.

Measure Your Performance By Topic

  • Export your GSC data and use a =VLOOKUP formula to match your clicks to your URLs.
  • Export your conversion data and use a =VLOOKUP formula to match your conversions (leads, sales, sign-ups, or revenue) to the right URL.
  • You can then copy your topics column onto a new sheet. Remove duplicates and use the =SUMIF formula to aggregate your click data and conversion data by topic.

Choose What Topics You’ll Be Translating First

Using this data, you can now choose what topics are most likely to drive conversions based on your core market data. Choose how many topics or pieces of content you’ll be translating based on your budget.

Personally, I like translating one topic at a time because I’ve found that generating topical authority on one specific topic makes it easier for me to rank on an adjacent topic that I write about next.

7. Localize Your English Content

Once you’re set up with all your key pages and a few content topics, it’s time to evaluate your investment and see where you could be getting a bigger return.

At this stage, many companies have translated their content into a few different languages and likely copied the US content into their UK and Australian sites. Now that you’ve done some translation, it’s time to work on localization.

If you’ve just copied your US content into your UK and Australian sites, your Google Search Console indexing report might be screaming at you, “Duplicate, Google selected a different canonical than the user.”

A very easy fix that could yield great returns is to localize your English content to the nuances of those English-speaking markets.

You will want to instruct your translation and localization providers to adapt the spellings of certain words, change the choice of words, introduce local expressions, and update any cited statistic for the US with their local equivalent.

For example, if I’m targeting a British audience, “analyze” becomes “analyse,” a “stroller” becomes a “pram,” and “soccer” becomes “football.”

8. Invest In In-market Content

Once you’ve got the basics in place, you can start tackling the specific needs of other markets. This strategy is expensive, and you should only use it in your priority markets, but it can really set you apart from your competitors.

For this, you will need to work with a local linguist to identify pain points, use cases, or needs exclusive to your target market.

For example, if France suddenly made it mandatory to run a diversity and inclusion study for companies with over 250 employees, I’d want to know this and create some content on DEI surveys at SurveyMonkey.

9. Integrate With Other Content Workflows

In step six, we evaluated our top-performing content, chose the best articles to translate, and got it all down. But wait. Some of these source articles have been updated. And there is even more content now!

To run a successful international SEO campaign you must integrate with all the other teams publishing content within your organization.

Usually, the teams creating content in an organization are SEO, content, PR, product marketing, demand generation, customer marketing, customer service, customer education, or solutions engineering.

That’s a lot, and you won’t be able to integrate with everyone all at once. Prioritize the teams that create the most revenue-generating content, such as SEO, content, or product marketing.

Working with these teams, you will have to establish a process for what happens when they create a new piece, update some content, or remove an existing piece.

These processes can differ for everyone, but I can tell you what I do with my team and hope it inspires you.

  • When a piece of content that’s already been localized into international markets is updated, we get the content in a queue to be re-localized the next quarter.
  • When they create a new piece of content, we evaluate its performance, and if it’s performing above average, we add it to a localization queue for the next quarter.
  • When they change the URL of a piece of content or delete it, all international sites must follow suit at the same time, since due to some technical limitations, not making the change globally would create some hreflang issues.

Wrapping Up

International SEO is vast and complex, and no article can cover it all, but many interesting resources have been created by SEO pros across the community for those who want to learn more.

Navigating the complexities of international SEO is no small feat. It’s an intricate dance of aligning commercial strategies with technical precision, cultural insights, and data-driven decisions.

From understanding your company’s core motives for global expansion to meticulously implementing hreflang tags and localizing content, every step plays a crucial role in building a successful international presence.

More resources: 


Featured Image: BritCats Studio/Shutterstock

New WordPress Plugin Simplifies Achieving Success via @sejournal, @martinibuster

The co-founders of Yoast have launched a plugin that helps users plan tasks, defeat procrastination, and remove distractions, making it easier to achieve success. This plugin simplifies managing critical tasks like maintaining website health, publishing posts, and updating content.

Why This Plugin Helps Users Become Successful

A reason why some websites fail to achieve all that they are capable of is momentum and consistent output. Creators to have a plan that is rigorously followed generally experience more success in search. Winning is fun but getting there is not always fun.

Immediate rewards are a powerful motivator for success. This new plugin makes achievement feel instantly gratifying, which is why it deserves serious consideration.

Clarity, Focus And Achievements

Working at home as a solopreneur or with remote workers can be challenging because there are so many distractions. People are generally task oriented but not necessarily hard-wired to follow a mental list of things to do. It’s easier when someone tells you what to do but the reality is that we have to take charge and tell ourselves what to do in order to achieve great things.

That’s the brilliant thing about the new Progress Planner plugin, it allows users to create a road map to success within the context of the WordPress site itself, embedded within the environment the user is working in.

One of the ingenious features of Progress Planner is that it gamifies task completion with badges that remind users of how much they’ve achieved, subtly encouraging them to continue completing tasks. It’s literally rewarding the brain with feedback on completion of a task, a mental pat on the back.

The Progress Planner website describes the tool like this:

“It simplifies website management by providing a clear overview of your tasks, tracking your progress, and keeping you motivated.”

Money’s a nice motivator but immediate positive feedback is a powerful motivator for progressing from achievement to achievement.

Progress Planner Beta

The plugin is currently in Beta, which is one step ahead of the Alpha stage where bugs are worked out. This means that the plugin has full functionality but is still collecting feedback from users. Nevertheless, Progress Planner is ready for use right now and the official launch date is set for October 3, 2024.

The plugin is 100% free to use and a pro version is planned for sometime in the future that will add even more features.

Progress Planner, by the co-founders of Yoast, is available right now from the official WordPress Plugin Repository and also in the plugin dashboard in the WordPress admin.

Read more and download the plugin: Progress Planner Plugin At WordPress.org

Visit the Progress Planner Website: Progress Planner

Featured Image by Shutterstock/Cast Of Thousands