8 Common Robots.txt Issues And How To Fix Them via @sejournal, @TaylorDanRW

Robots.txt is a useful and powerful tool to instruct search engine crawlers on how you want them to crawl your website. Managing this file is a key component of good technical SEO.

It is not all-powerful – in Google’s own words, “it is not a mechanism for keeping a web page out of Google” – but it can help prevent your site or server from being overloaded by crawler requests.

If you have this crawl block on your site, you must be certain it’s being used properly.

This is particularly important if you use dynamic URLs or other methods that generate a theoretically infinite number of pages.

In this guide, we will look at some of the most common issues with the robots.txt file, their impact on your website and your search presence, and how to fix these issues if you think they have occurred.

But first, let’s take a quick look at robots.txt and its alternatives.

What Is Robots.txt?

Robots.txt uses a plain text file format and is placed in the root directory of your website.

It must be in the topmost directory of your site. Search engines will simply ignore it if you place it in a subdirectory.

Despite its great power, robots.txt is often a relatively simple document and a basic robots.txt file can be created in seconds using an editor like Notepad. You can have fun with them and add additional messaging for users to find.

8 Common Robots.txt Issues And How To Fix ThemImage from author, February 2024

There are other ways to achieve some of the same goals that robots.txt is usually used for.

Individual pages can include a robots meta tag within the page code itself.

You can also use the X-Robots-Tag HTTP header to influence how (and whether) content is shown in search results.

What Can Robots.txt Do?

Robots.txt can achieve a variety of results across a range of different content types:

Webpages can be blocked from being crawled.

They may still appear in search results, but they will not have a text description. Non-HTML content on the page will not be crawled either.

Media files can be blocked from appearing in Google search results.

This includes images, video, and audio files.

If the file is public, it will still “exist” online and can be viewed and linked to, but this private content will not show in Google searches.

Resource files like unimportant external scripts can be blocked.

But this means if Google crawls a page that requires that resource to load, the Googlebot robot will “see” a version of the page as if that resource did not exist, which may affect indexing.

You cannot use robots.txt to completely block a webpage from appearing in Google’s search results.

To achieve that, you must use an alternative method, such as adding a noindex meta tag to the head of the page.

How Dangerous Are Robots.txt Mistakes?

A mistake in robots.txt can have unintended consequences, but it’s often not the end of the world.

The good news is that by fixing your robots.txt file, you can recover from any errors quickly and (usually) in full.

Google’s guidance to web developers says this on the subject of robots.txt mistakes:

“Web crawlers are generally very flexible and typically will not be swayed by minor mistakes in the robots.txt file. In general, the worst that can happen is that incorrect [or] unsupported directives will be ignored.

Bear in mind though that Google can’t read minds when interpreting a robots.txt file; we have to interpret the robots.txt file we fetched. That said, if you are aware of problems in your robots.txt file, they’re usually easy to fix.”

8 Common Robots.txt Mistakes

  1. Robots.txt Not In The Root Directory.
  2. Poor Use Of Wildcards.
  3. Noindex In Robots.txt.
  4. Blocked Scripts And Stylesheets.
  5. No Sitemap URL.
  6. Access To Development Sites.
  7. Using Absolute URLs.
  8. Deprecated & Unsupported Elements.

If your website behaves strangely in the search results, your robots.txt file is a good place to look for any mistakes, syntax errors, and overreaching rules.

Let’s take a look at each of the above mistakes in more detail and see how to ensure you have a valid robots.txt file.

1. Robots.txt Not In The Root Directory

Search robots can only discover the file if it’s in your root folder.

That’s why there should be only a forward slash between the .com (or equivalent domain) of your website, and the ‘robots.txt’ filename, in the URL of your robots.txt file.

If there’s a subfolder in there, your robots.txt file is probably not visible to the search robots, and your website is probably behaving as if there was no robots.txt file at all.

To fix this issue, move your robots.txt file to your root directory.

It’s worth noting that this will need you to have root access to your server.

Some content management systems will upload files to a “media” subdirectory (or something similar) by default, so you might need to circumvent this to get your robots.txt file in the right place.

2. Poor Use Of Wildcards

Robots.txt supports two wildcard characters:

  • Asterisk (*) – represents any instances of a valid character, like a Joker in a deck of cards.
  • Dollar sign ($) – denotes the end of a URL, allowing you to apply rules only to the final part of the URL, such as the filetype extension.

It’s sensible to adopt a minimalist approach to using wildcards, as they have the potential to apply restrictions to a much broader portion of your website.

It’s also relatively easy to end up blocking robot access from your entire site with a poorly placed asterisk.

Test your wildcard rules using a robots.txt testing tool to ensure they behave as expected. Be cautious with wildcard usage to prevent accidentally blocking or allowing too much.

3. Noindex In Robots.txt

This one is more common on websites that are over a few years old.

Google has stopped obeying noindex rules in robots.txt files as of September 1, 2019.

If your robots.txt file was created before that date or contains noindex instructions, you will likely see those pages indexed in Google’s search results.

The solution to this problem is to implement an alternative “noindex” method.

One option is the robots meta tag, which you can add to the head of any webpage you want to prevent Google from indexing.

4. Blocked Scripts And Stylesheets

It might seem logical to block crawler access to external JavaScripts and cascading stylesheets (CSS).

However, remember that Googlebot needs access to CSS and JS files to “see” your HTML and PHP pages correctly.

If your pages are behaving oddly in Google’s results, or it looks like Google is not seeing them correctly, check whether you are blocking crawler access to required external files.

A simple solution to this is to remove the line from your robots.txt file that is blocking access.

Or, if you have some files you do need to block, insert an exception that restores access to the necessary CSS and JavaScript.

5. No XML Sitemap URL

This is more about SEO than anything else.

You can include the URL of your XML sitemap in your robots.txt file.

Because this is the first place Googlebot looks when it crawls your website, this gives the crawler a headstart in knowing the structure and main pages of your site.

While this is not strictly an error – as omitting a sitemap should not negatively affect the actual core functionality and appearance of your website in the search results – it’s still worth adding your sitemap URL to robots.txt if you want to give your SEO efforts a boost.

6. Access To Development Sites

Blocking crawlers from your live website is a no-no, but so is allowing them to crawl and index your pages that are still under development.

It’s best practice to add a disallow instruction to the robots.txt file of a website under construction so the general public doesn’t see it until it’s finished.

Equally, it’s crucial to remove the disallow instruction when you launch a completed website.

Forgetting to remove this line from robots.txt is one of the most common mistakes among web developers; it can stop your entire website from being crawled and indexed correctly.

If your development site seems to be receiving real-world traffic, or your recently launched website is not performing at all well in search, look for a universal user agent disallow rule in your robots.txt file:

User-Agent: *


Disallow: /

If you see this when you shouldn’t (or don’t see it when you should), make the necessary changes to your robots.txt file and check that your website’s search appearance updates accordingly.

7. Using Absolute URLs

While using absolute URLs in things like canonicals and hreflang is best practice, for URLs in the robots.txt, the inverse is true.

Using relative paths in the robots.txt file is the recommended approach for indicating which parts of a site should not be accessed by crawlers.

This is detailed in Google’s robots.txt documentation, which states:

A directory or page, relative to the root domain, that may be crawled by the user agent just mentioned.

When you use an absolute URL, there’s no guarantee that crawlers will interpret it as intended and that the disallow/allow rule will be followed.

8. Deprecated & Unsupported Elements

While the guidelines for robots.txt files haven’t changed much over the years, two elements that are oftentimes included are:

  • Crawl-delay.
  • Noindex.

While Bing supports crawl-delay, Google doesn’t, but it is often specified by webmasters. You used to be able to set crawl settings in Google Search Console, but this was removed towards the end of 2023.

Google announced it would stop supporting the noindex directive in robots.txt files in July 2019. Before this date, webmasters were able to use the noindex directive in their robots.txt file.

This was not a widely supported or standardized practice, and the preferred method for noindex was to use on-page robots, or x-robots measures at a page level.

How To Recover From A Robots.txt Error

If a mistake in robots.txt has unwanted effects on your website’s search appearance, the first step is to correct robots.txt and verify that the new rules have the desired effect.

Some SEO crawling tools can help so you don’t have to wait for the search engines to crawl your site next.

When you are confident that robots.txt is behaving as desired, you can try to get your site re-crawled as soon as possible.

Platforms like Google Search Console and Bing Webmaster Tools can help.

Submit an updated sitemap and request a re-crawl of any pages that have been inappropriately delisted.

Unfortunately, you are at the whim of Googlebot – there’s no guarantee as to how long it might take for any missing pages to reappear in the Google search index.

All you can do is take the correct action to minimize that time as much as possible and keep checking until Googlebot implements the fixed robots.txt.

Final Thoughts

Where robots.txt errors are concerned, prevention is always better than the cure.

On a large revenue-generating website, a stray wildcard that removes your entire website from Google can have an immediate impact on earnings.

Edits to robots.txt should be made carefully by experienced developers, double-checked, and – where appropriate – subject to a second opinion.

If possible, test in a sandbox editor before pushing live on your real-world server to avoid inadvertently creating availability issues.

Remember, when the worst happens, it’s important not to panic.

Diagnose the problem, make the necessary repairs to robots.txt, and resubmit your sitemap for a new crawl.

Your place in the search rankings will hopefully be restored within a matter of days.

More resources:


Featured Image: M-SUR/Shutterstock

Top 3 SEO Checklists For On-Page & Technical SEO In 2024 via @sejournal, @WixStudio

This post was sponsored by Wix. The opinions expressed in this article are the sponsor’s own.

In the last 18 months, conversations about SEO have been dominated by artificial intelligence.

New ways of working and new user journeys for search visitors have come to the forefront.

In 2024, we can clearly see that there’s a great mix of “old hat” and “new hat” SEO best practices that you can start implementing right now.

To help you keep up with the current SEO best practices, we’ve created a series of:

Why Use SEO Checklists?

Doing SEO is often a bit like conducting a symphony.

There are lots of SEO activities, big and small, taking place across each website, and each SEO task, whether it be on-page or technical, contributes to the larger success of your website.

Just as a conductor has their sheet music, the best consultants and agencies use SEO checklists, templates, cheat sheets, and other project management documents to make sure everything is on track.

The medium of the SEO checklist has evolved from being as simple as a few items scribbled on the back of an envelope to something as complex as a dynamically generated interface within your CMS, offering AI functionality and integration with your favorite tools.

This versatility means that they can be reworked to suit most requirements.

In the sake of versatility, here are three, high-level types of SEO checklists that can help you elevate your brand’s website.

1. SEO Checklists On Shared Documents

If you’ve been in the industry for a while, you’re probably familiar with using a spreadsheet to keep track of SEO progress.

Now, we’re living in the fast lane, with the ability to leverage shared SEO. Easily transferrable, this can be as complex or simple as required.

Downloadable SEO checklists & templatesImage created by Wix, February 2024

2. Dynamic, Real-Time SEO Checklists

Fast forward to a next-generation SEO checklist: The Dynamic SEO Checklist.

A dynamic SEO checklist responds to updates in content in real-time.

This means that anytime a change is made to your website, you get a series of next steps to execute to make your content rank.

Examples Of Real-Time SEO Checklists

Site-level SEO Assistant is one new example of a real-time checklist.

This type of checklist is built directly into your CMS and includes entry points and integrations that allow users to complete tasks in the list.

From setting indexability to connecting to Google Search Console, you can complete important tasks from within the list itself.

With intelligent recommendations and the ability to skip tasks as necessary, this kind of checklist works in tandem with your website.

Built-in functionality like this can help you save time and get the most out of a CMS.

Top 3 SEO Checklists For On-Page & Technical SEO In 2024Experience created by Wix, Feburary 2024

3. AI-Powered SEO Checklists

With the assistance of AI, you can get a dynamically generated checklist to help you execute your SEO workload.

Plus, this type of list can also help you carry out within the list itself, simply by using AI inputs.

For instance, the checklist in the Wix SEO Assistant uses a keyword prompt to create AI meta descriptions and title tags for a page.

Once these are generated, you can add and tick them off your list, right in the CMS.

Top 3 SEO Checklists For On-Page & Technical SEO In 2024Experience created by Wix, February 2024

Looking at user data, we see that the addition of AI to a checklist can increase usability significantly.

How To Properly Use An SEO Checklist For Success

If you aren’t already using an SEO checklist to manage your workload, we urge you to try any of the above styles of checklist.

Increase Your Agency’s Profitability

If you’re an agency owner or account manager, checklists are particularly efficient because they can be reused and adapted for each client.

This can save time on research and development (R&D) and become part of the corpus of resources you make available to clients and staff.

Checklists for SEO tasks can also assist with more efficient onboarding for new clients and projects, and help you expand your offering and improve client satisfaction.

When managing clients, tools like the Wix Studio Client Kit mean that account managers can upload client checklists, templates, and even video tutorials directly into the dashboard of the CMS.

This means that materials for client success are accessible to everyone working on the site, at all times.

Top 3 SEO Checklists For On-Page & Technical SEO In 2024Image created by Wix, February 2024

If you’re managing SEO teams, creating frameworks for SEO account management can help you gain and retain clients for the long term.

Joshua George, founder of Clickslice Agency explains that he developed his SEO project proposal template “after testing 20 different variants of proposals,”  but once he “finally found one which converts the best,” he was able to get better results from client pitches.

Similarly,  freelance SEO, Nick LeRoy, developed a client retention checklist to manage touch points and habits that build client trust.

Top 3 SEO Checklists For On-Page & Technical SEO In 2024Image created by Wix, February 2024

Better efficiency for SEO agencies and freelancers means more opportunities to deliver high-value outcomes and increase profitability – a checklist can be an important part of that process.

Make Knowledge More Accessible Across Your Team

Templates and checklists are also a highly accessible way to hand off to juniors and even clients easily.

As well as being designed to help your team accomplish goals, checklists can also serve as a knowledge base to help your team grow. This approach can help you get drive visibility on the SERP.

Top 3 SEO Checklists For On-Page & Technical SEO In 2024Image created by Wix, February 2024

Founder of AS Marketing agency, Adriana Stein, explains that she uses a blog writing template because, “when you use a particular structure, provide helpful information, as well as follow SEO best practices, that gives your content the best possible chance to rank, drive organic traffic, and when scaled out, drive conversions.”With live collaborative documents, teams can adapt lists to changing requirements and ensure that the knowledge base of the team continues to grow, even if account managers change roles.

Improve Quality Assurance

It is often assumed that it is beginner-level SEO specialists who use checklists, but this is not exactly the case.

For more seasoned SEOs, checklists often serve as a means of maintaining the quality of implementation when working with multiple teams and stakeholders. Experienced SEO professionals create checklists to ensure consistency of delivery when handing off day-to-day tasks to other members of their teams.

Ashwin Balakrishnan, Head of Marketing at Optmyzr, explains that for tasks like backlink tracking, he creates templated documents to reduce errors and ensure a consistent level of implementation.

Balakrishnan says that, “transferring knowledge and experience to junior SEOs (especially teammates) goes a lot smoother for everyone when you use checklists and templates. It gives newer SEOs confidence and frameworks to work off, and it allows more seasoned practitioners to review work faster and avoid giving arbitrary feedback.”

This can free up time for senior SEOs to pursue more complex SEO, billable tasks.

So rather than dying a death, the humble SEO checklist has evolved.

What’s Next For SEO Checklists?

Given the wealth of data sets, data feeds, automation, and AI tools available to marketers, the active integration of AI into SEO should only continue. But as well as being able to generate inputs, we are likely to see data-driven, AI-crafted guidance from checklists become more common.

Those who use checklists in the format of Google Sheets and MS Excel will benefit from generative native apps and enhancements in the AI workspace race. Being able to harness scripts, feeds, and add-ons is nothing new, but AI offers new scalability for task completion, offering new opportunities for growth but the value of the SEO checklist is alive and well in 2024.

Download checklists, templates, and toolkits to develop smoother SEO processes and ramp up productivity—on any project.


Image Credits

Featured Image: Image by Wix. Used with permission.

Are Rich Results Dead? Maximize The Semantic Value Of Schema Markup via @sejournal, @marthavanberkel

Last year was the year of change for search, especially for schema markup.

We saw a significant loss in clicks due to changes made to certain rich results, while simultaneously witnessing the most number of new rich results introduced by Google in a year.

In August, I even asked myself, are rich results dead? Good news – the answer is a resounding no!

Not only are we seeing the rich results landscape continue to evolve, but the value of schema markup is now more than just rich results.

It is a data layer that enables semantic understanding for AI-powered search engines and experiences.

What Is Schema Markup?

Schema markup, also known as structured data, is data added to your website using the descriptive Schema.org vocabulary.

By applying schema markup, content creators can clearly articulate the intent and meaning of their content, enabling search engines to better understand the content on their web pages.

Adding defined required and recommended schema markup to a webpage can also make it eligible for an enhanced search result, known as a rich result.

Rich results include reviews, ratings, course info, events, local business, and more.

example of a product rich resultScreenshot from Google, January 2024

Rich results tend to drive a higher click-through rate from the search engine result page (SERP), resulting in more user engagement.

In 2023, product, review, and FAQ rich results were the top awarded rich results for our customers.

However, we know that the search landscape is unpredictable.

Rich results can come and go at Google’s whim.

Generative AI search engines are disrupting the way we consume information, and the search engines are iterating on the search experience as a result.

This raises the question of whether rich results, the coveted outcome for implementing schema markup, are becoming obsolete.

Are Rich Results Dead?

In the first half of 2023, we saw OpenAI, Bing, and Google introduce their generative AI search engines to the world.

This was a major disruption in the world of search, as these generative AI search engines would present answers in a multi-modal format to users directly in the SERP, giving answers in the search engine.

This was a similar experience to zero-click searches in 2020 but on an even bigger stage. It was clear that big changes were coming.

Then, on August 8, 2023, Google announced that it would be reducing the frequency of FAQ rich results being displayed on the SERP and deprecating HowTo rich results entirely.

FAQ rich results would now be exclusively available for “well-known, authoritative government and health websites.”

Many organizations that previously leveraged FAQ rich results to stand out on the SERP were no longer awarded these rich results. This included several reputable healthcare organizations.

Then, on September 14, 2023, all FAQ rich results were gone from the SERP.

Prior to this change, FAQ rich results were one of the most popular rich results because they enhanced visibility for any content type content and also delighted users with answers directly in the SERP.

Many of the other rich results had specific content requirements (reviews, ratings, video, etc), making them less versatile than FAQ.

Without FAQ and HowTo rich results, the types of content eligible for rich results were greatly reduced. Was this the beginning of the end for rich results?

Don’t Worry; Rich Results Are Not Dead

Google answered the question of whether rich results are dead in October 2023, when it released a new episode of Google Search News.

In the episode, John Mueller stated that new rich results would be added while others may disappear over time.

Staying true to its word, in the last quarter of 2023, Google introduced six new rich results – the most it has ever released in a year.

What can we learn from these new rich results? 

First, they are becoming very industry-specific.

Of the six new rich results released in 2023, only two were ‘general’ rich results like FAQ rich results that organizations leverage.

Second, many of the rich results are complimenting previous data feeds.

This may be the start of Google’s more programmatic consumption of structured data, as mentioned by Ryan Levering in April 2022’s Search Off the Record podcast.

Why Are Rich Results & Schema Markup Important To Google And Bing?

Rich results improve a user’s search experience. However, there is a bigger reason why Google continues to invest in them.

Schema Markup Provides Context About Your Content

As John Mueller pointed out in his Search Update, schema markup is machine-readable code you can add to your web pages to help search engines better understand your content.

Google uses large language models to understand the content on a page, but LLMs are prone to hallucinations and misinterpretations.

Furthermore, the things or entities mentioned in your content can be ambiguous.

For example, the word “Apple” is both a fruit and a brand. Without additional context, it can be hard for the LLM to disambiguate which Apple you’re referring to.

Schema markup can overcome this by providing additional information and context about the entities mentioned in your content.

This makes the content easier to understand and increases the machine’s confidence in the data, allowing it to provide more accurate and relevant search results.

When you implement schema markup on a page, you aren’t just defining the entities on that page. You’re also describing their relationship to other entities on your site and on the web.

This interconnected web of information about your website forms a structured content knowledge graph.

Schema Markup Allows You To Control How Your Content Is Understood

Your content knowledge graph enables search engines to understand and infer new knowledge about your organization.

Generative AI search engines like Gemini, SGE, and ChatGPT benefit significantly from a well-constructed knowledge graph, as shown in the image below.

Summarization of the pros and cons for LLMs and KGsImage from arxiv.org, January 2024

By providing factual and connected schema markup in a graph, your website has now become a data layer for machines to consume, giving you a control point on how you want your data understood and enabling them to make accurate inferences about your brand.

Start Maximizing The Semantic Value of Schema Markup

Rich results are not dead. In fact, they have never been more alive!

Google uses the schema markup on your site to understand your content while helping you stand out in the SERP, creating an engaging search experience for your consumers.

Google will continue releasing new rich results in exchange for your data because their AI-powered search experiences depend on access to your data to deliver accurate, relevant answers.

However, the value of schema markup extends far beyond achieving rich results.

By implementing robust, connected schema markup, you are essentially creating a data layer that allows you to control how search engines and machines consume and understand your content.

That way, the LLMs that power search experiences like SGE, ChatGPT, Gemini, and the new Bing can make accurate inferences about your content, preparing your organization for future search experiences.

One could argue that schema markup has never been more strategic for companies aiming to maintain a control point in preparation for AI.

More resources: 


Featured Image: Gracia Chua/Schema App

Google and EEAT, Explained

Google hires thousands of personnel to view and evaluate web pages for organic search rankings. A key metric for those human raters is “Experience, Expertise, Authoritativeness, and Trustworthiness.” Google emphasizes pages that demonstrate those qualities.

What’s unclear, however, is how Google integrates EEAT into its algorithm. Google executives have shared conflicting views.

In 2022, Google’s vice president of search, Hyung-Jin Kim, stated that EAT was a “core part.” (Google originally coined the metric as “Expertise, Authoritativeness, and Trustworthiness.” In December 2022, it added “Experience,” morphing “EAT” to “EEAT.”)

In 2024, Google’s search liaison, Danny Sullivan, tweeted it had never been a ranking factor.

Google’s “Search Central Blog” states:

Search raters have no control over how pages rank. Rater data is not used directly in our ranking algorithms. Rather, we use them as a restaurant might get feedback cards from diners. The feedback helps us know if our systems seem to be working.

Hence EEAT is for Google’s quality control. But it’s also for creators. The blog post adds, “… the guidelines may help you self-assess how your content is doing from an E-E-A-T perspective, improvements to consider… .”

The bottom line is Google wants to rank web pages with strong EEAT. The guidelines (PDF) include:

  • Expertise of the author. Add details such as work history.
  • Research methods. Explain how the content was created. For example, for product reviews, state how many you tested and how.
  • Purpose. Address the rationale for publishing the content. It’s a vague suggestion, although it reinforces the goal of helping humans, not search engines. Potential reasons might include client interactions, outside commentary, new data, and more.

EEAT applies to informational content, not product or category pages. Moreover, Google scrutinizes content that impacts “the health, financial stability, or safety of people, or the welfare or well-being of society.” Google calls such content “Your Money or Your Life” — YMYL.

Google tells its raters that YMYL pages require much expertise and authority. So if you publish financial or health advice, closely follow the guidelines.

EEAT in Brief

Here’s my interpretation of EEAT in practice. The underlying principles existed long before the acronym. Many are more or less common sense: Ensure your content is trustworthy, thorough, authentic, and helpful to humans.

Content Type EEAT Inclusions
Health topics Detailed author info to demonstrate expertise. Include citations in the content to (i) professional research, (ii) opinions of doctors and scientists, and (iii) reputable publications.
Financial or legal advice Detailed author info to demonstrate expertise. Include citations in the content to (i) opinions of accountants and lawyers, (ii) professional organizations, and (iii) reputable publications.
News reporting Accurate factual information; expertise of experienced journalists; quotations from experts on the topic.
Legislation, public policy, societal concerns Official government sources; reputable media sources; trusted and varied opinions; independent research.
News or advice related to children Views and recommendations of professionals (doctors, educators, counselors); professional publications and organizations; independent research.
Product reviews. The writer’s first-hand experience, including the number of products tested and the process and timeframe.

Instant SEO Boost: 10 Easy Strategies That Make An Immediate Impact via @sejournal, @hethr_campbell

If you find yourself struggling to prioritize your SEO tasks effectively, you’re certainly not alone. 

It can be overwhelming trying to determine where you should focus your efforts to raise your website’s rankings.

So rather than drowning in a sea of meta descriptions, why not focus on quick optimizations that can deliver immediate results?

If you’re looking for quick wins you can apply to your SEO strategy today, tune into our upcoming webinar

On March 6, we’re teaming up with Ahrefs to bring you 10 powerful strategies to skyrocket your site’s visibility and win the hearts of your clients.

In this live webinar, we’ll guide you through a series of actionable steps that take mere seconds or minutes to implement but yield significant improvements in your website’s rankings. 

You’ll learn how to:  

  • Update existing pages for low-hanging fruit keywords.
  • Find featured snippet opportunities.
  • Update pages with declining traffic in the past 6 months.
  • Uncover content gaps between you and your top competitors.
  • Republish old pages with low traffic.
  • Redirect links from most linked-to broken pages.
  • Get internal linking suggestions.
  • And more!

Andrei Țiț, Product Marketer at Ahrefs will walk you through how to prioritize the SEO tasks with the highest impact, as well as the tools you can use to complete them.

We’ll also show you how to streamline your link building tactics, uncover valuable content opportunities, and prioritize technical issues to ensure maximum impact on your SEO efforts.

Plus, be sure to stick around for a live Q&A session with Andrei, where you can get your questions answered in real-time! 

Register now and discover how you can start delivering quick wins without the hassle.

Whether you’re a beginner or an experienced SEO pro, these strategies will help you focus on the right tasks and achieve tangible results.

Don’t miss out! Join us from anywhere in the world and learn the skills you need to take your website’s rankings to new heights!

What Data Science Can Do for Site Architectures via @sejournal, @artios_io

The past decade has marked the shift of SEO from spreadsheet-driven, anecdotal best practices to a more data-driven approach, evidenced by the greater numbers of SEO pros learning Python.

As Google’s updates increase in number (11 in 2023), SEO professionals are recognizing the need to take a more data-driven approach to SEO, and internal link structures for site architectures are no exception.

In a previous article, I outlined how internal linking could be more data-driven, providing Python code on how to evaluate the site architecture statistically.

Beyond Python, data science can help SEO professionals more effectively uncover hidden patterns and key insights to help signal to search engines the priority of content within a website.

Data science is the intersection of coding, math, and domain knowledge, where the domain, in our case, is SEO.

So while math and coding (invariably in Python) are important, SEO is by no means diminished in its importance, as asking the right questions of the data and having the instinctive feel of whether the numbers “look right” are incredibly important.

Align Site Architecture To Support Underlinked Content

Many sites are built like a Christmas tree, with the home page at the very top (being the most important) and other pages in descending order of importance in subsequent levels.

For the SEO scientists among you, you’ll want to know what the distribution of links is from different views. This can be visualized using the Python code from the previous article in several ways, including:

  • Site depth.
  • Content type.
  • Internal Page Rank.
  • Conversion Value/Revenue.
Internal links to URLImage by author, December 2023

The boxplot effectively shows how many links are “normal” for a given website at different site levels. The blue boxes represent the interquartile range (i.e., the 25th and 75th quantiles) which is where most (67% to be precise) of the number of inbound internal links lie.

Think of the bell curve, but instead of viewing it from the side (as you would a mountain), you’re viewing it like a bird flying overhead.

For example, the chart shows that for pages that are two levels down from the home page, the blue box indicates that 67% of URLs have between five and nine inbound internal links. We can also see this is considerably (and perhaps unsurprisingly) much lower than pages that are one hop away from the home page.

The thick line that cuts the blue box is the median (50th quantile), representing the middle value. Going with the above example, the median inbound internal links are 7 for site level 2 pages, which is about 5,000 times less than those in site level 1!

On a side note, you may notice that the median line isn’t visible for all blue boxes, the reason being the data is skewed (i.e., not normally distributed like a bell-shaped curve).

Is This Good? Is This Bad? Should SEO Pros Be Worried?

A data scientist with no knowledge of SEO might decide that it’d be better to redress the balance by working out the distribution of internal links to pages by site level.

From there, any pages that are, say, below the median or the 20th percentile (quantile in data science speak) for their given site level, a data scientist might conclude that these pages require more internal links.

As such, this often means that pages that share the same number of hops from the home page (i.e., same site depth level) are of equal importance.

However, from a search value perspective, this is unlikely to be true, especially when you consider that some pages on the same level simply have more search demand than others.

Thus, the site architecture should prioritize those pages with more search demand than those with less demand regardless of their default place in the hierarchy – whatever their level!

Revising True Internal Page Rank (TIPR)

True Internal Page Rank (TIPR), as popularised by Kevin Indig, has taken a rather more sensible approach by incorporating the external PageRank, i.e., earned from backlinks. In simple maths terms:

TIPR = Internal Page Rank x Page Level Authority of Backlinks

Although the above is the non-scientific version of his metric, it’s nonetheless a much more useful and empirical way of modeling what is the normal value of a page’s value within a website architecture. If you’d like the code to compute this, please see here.

Furthermore, rather than applying this metric to site levels, it’s far more instructive to apply this by content type. For an ecommerce client, we see the distribution of TIPR by content type below:

True internal page rank Image by author, December 2023

The plot in this online store’s case is that the median TIPR for categories content or Product Listing Pages (PLPs) is about two TIPR points.

Admittedly, TIPR is a bit abstract, as how does that translate to the amount of internal links required? It doesn’t – at least not directly.

Abstraction notwithstanding, this is still a more effective construct for shaping site architecture.

If you wanted to see which categories were underperforming for their rank position potential, you’d simply see that PLP URLs were below the 25th quantile and perhaps look for internal links from pages of a higher TIPR value.

How many links and what TIPR? With some modeling, that’s an answer for another post.

Introducing Revenue Internal Page Rank (RIPR)

The other important question worth answering is: which content deserves higher rank positions?

Kevin also advocated a more enlightened approach to align internal link structures towards conversion values, which many of you are hopefully already applying to your clients; I must heartily agree.

A simple non-scientific solution is to take the ratio of the ecommerce revenue to the TIPR i.e.

RIPR = Revenue / TIPR

The above metric helps us see what normal revenue per page authority looks like, as visualized below:

revenue internal page rank Image by author, December 2023

As we can see, the picture changes somewhat; suddenly, we see no box (i.e., distribution) for blog content because no revenue is recorded against that content.

Practical applications? If we use this as a model by content type, any pages that are higher than the 75th quantile (i.e., north of their blue box) for their respective content type should have more internal links added to them.

Why? Because they have high revenue but are very low in Page Authority, meaning they have a very high RIPR and should therefore be given more internal links to get it closer to the median.

By contrast, those with lower revenue but too many significant internal links will have a lower RIPR and should thus have links taken away from them to allow the higher revenue content to be assigned more importance by the search engines.

A Caveat

RIPR has some assumptions built in, such as analytics revenue tracking being set up properly so that your model forms the basis for effective internal link recommendations.

Of course, as in TIPR, one should model what an internal link is worth in terms of how much RIPR an internal link is worth from any given page.

That’s before we even get to the location of the internal link placement itself.

More resources: 


Featured Image: NicoElNino/Shutterstock

A successful link building strategy in 6 steps

SEO is all about creating a great experience for people visiting your website, or seeing your snippet in the search results. That’s why a lot of your SEO efforts will likely be focused on things like content, UX and pagespeed. Efforts that help you optimize your page or website overall. As they should be! They help you become the best search result for search engines and people alike. However, there are also a few things outside of your website that you should not forget about. An important one being link building. In this post, we’ll discuss the 6 steps to create a successful link building strategy for any business.

In a nutshell, link building is the act of getting other websites to link to your page. The reason why you should invest some time into this is because these links are important for SEO. They help your pages rank higher in the search results. But that’s not the only thing they do:

A good link, first and foremost, drives traffic to your site.

Other websites linking to your page will help in getting more (referral) traffic to your site. This grows your overall traffic and brings new people into contact with your website. To make sure these efforts aren’t in vain, it’s good to think about which websites would make sense in linking to your pages. You want people who are visiting that website to also enjoy yours when they’re being directed there, but we’ll go into that further one.

An important note is that link building should always be done from a holistic SEO perspective. Meaning that you should focus on getting quality backlinks that will actually generate traffic for your website. In addition, it means staying away from bad practices as these can hurt your rankings.

1. Really get to know your audience

To get an idea of what websites would ideally link to yours, you first need to have insights into who your audience is. You might have a hunch of who your audience is, but you’ll be surprised how often that doesn’t match with reality. Make sure to do some research and analyze your audience to get to know them. This can help you retain your current audience and perhaps even reach new audiences that might be interested in what you’re offering. All of this also gives you a better idea of other websites they might be interested in. If it feels natural, you can even ask them about other websites that they frequent.

To give you an example, say you run an online store with loads of craft materials. and blog about new craft ideas now and then. User research could give you the insight that a large part of your audience is parents looking for fun crafts to do with their kids. This could give you the idea to reach out to websites listing fun activities to do with kids. Something you wouldn’t have done if you did not know about that segment of your target audience. Or to give an example closer to home, at Yoast, we started with an audience that consisted mainly of WordPress developers. However, we wanted to broaden our audience to a more general group of WordPress users without losing our initial audience. So we created additional content that caters to our new audience and went to work to get links from other websites where these people can be found.

2. Create a list of sites that your audience visits

When you know more about your audience and, you can create a list of websites that will help you reach them. Use your research to find the websites that appeal to these people. Because links from these websites can help you reach your audience, especially if they don’t know about your website yet. Do pay attention whether there’s a logical connection to be made between you and a website on the list. This increases your chances of getting the link and is better for SEO as well. A link from a website that has absolutely nothing to do with your niche is not valuable when it comes to your position in the search results.

In addition, I want to note that a link from a spammy website is also not going to do you any good. These links can even backfire and hurt your rankings as Google is absolutely not a fan of bad link building practices. So stay away from spammy websites, paying for your links and other link building DON’Ts. Link building isn’t just a trick or something you can throw money at. Getting these links should feel like a normal marketing effort and part of a holistic SEO approach.

3. Write great content

To get other websites to link to your content, you need to have content that makes them want to link to your content. Which means that you need to create quality content. Content that appeals to your audience, is helpful and that showcases your unique point of view. If you sell products or services, don’t just write about why they’re awesome and why they should buy them. Write content that answers a question that your audience has or solves a problem they’re facing. Don’t center it around your product or service. Make your content genuinely helpful and show your expertise on the topic. This will not only build trust and authority, but you’ll also get more links to your page as other websites will see the value of your content.

To give an example: Let’s say you sell garden tools like lawnmowers. Instead of writing a blog post on why your lawnmower is the best one out there, write blog posts on topics like ‘How to get your garden ready for the summer’. Or if you sell furniture, write an article on the latest trends in interior design. This type of content is a lot more shareable for other people outside your company. Which will increase your chances of getting relevant links. If you need some guidance on creating quality content, we have lots of blog posts on content SEO and an SEO copywriting training course.

Guest blogging

When you’re a blogger or (aiming to be) an expert in your field, another option is guest blogging. This not only builds your authority, it’s a great way to get more links to your website. Often enough, blogs are looking for input from fellow bloggers and in return will let you link to your own content. This gets you a great link to your website and the opportunity to bring yourself into contact with their audience. Growing your reach.

Once again, be tactical in the websites you choose to partner up with. They should be trustworthy and relevant to your own website to get any real value out of it. Visit your favorite blogs, or blogs that are similar to yours, and check to see if they’re open to guest submissions. They’ll usually mention this on their contact or collaboration page.

4. Match content to the right website

When you’re happy with the content you’ve written, it’s time to dive into the list of websites you’ve made during the 2nd step. Which websites on there will be likely to link to which piece of content? You may be tempted to just send everything that you’ve created to everyone, but this will hurt your chances of anyone linking to your content. You will come across as spammy and you can’t ask those people to read 5 blog posts and decide which one they like.

Also good to know: if you have a long tail keyword approach (writing about small and niche subjects) the number of websites that are a good fit will be smaller. This isn’t a bad thing, as very specific content can mean that this smaller number of websites is more willing to link to your content. It can also mean that readers will appreciate your content even more, as there’s less of the same content out there and they’ll be eager to know more about your business.

5. Reach out in a personal way

After figuring out which content to send to whom, it’s time to reach out to them. You can always send an email, but social media like X/Twitter or LinkedIn are also a great way to contact people directly. To increase your chances of getting their attention, make sure to do your homework. Use your audience research and what you know about the website to personalize your message. Never send out automated emails or direct messages. Send them a polite message in which you tell them about your content, why you feel it would be of value for their audience and request them to place a link to your content. Please note that often, you will not get a reply at all.

To improve your chances, you need to explain why your content is unique. Trying to get a link for a very general blog post that could’ve been written by anyone, is less likely to succeed than unique content. Content that people can only find on your website. That being said, don’t make your message too long, as this will result in people not reading it at all. If you’re not sure where to start, you can also contact your business partners. They will probably be active in a field that’s related to yours and they’ll be more willing to link to your site (as you already know them personally). Just make sure that the backlink is relevant and doesn’t feel forced. Like I said before, link building should always feel natural.

6. Extend your reach through social media

Reaching out to specific people or websites is one way to get links to your content. Another option is to share your content on social media. This can also lead to other people sharing your content, which helps you extend your reach. And a wider reach gives you a higher chance of people linking to you on their websites as well. When people like, share and talk about your content on social media, you’re bound to reach new audiences and receive some more links as well.

A successful link building strategy should always be aimed at getting new people into contact them with your website. A (welcome) side effect of proper link building is a higher ranking in Google. As long as you consider link building as a way to reach out to other sites to get more visitors that will genuinely enjoy your content, you’re doing it right. Lots of luck!

Read more: Link building from a holistic SEO perspective »

Coming up next!

Google’s Core Update vs. Helpful Content

Google’s algorithm updates used to be straightforward. If rankings were impacted, search engine optimizers knew the cause. For example, the Panda update in 2011 affected sites with low-value content. The Penguin update in 2012 addressed spammy links.

But those causes are no longer apparent. Google’s explanations of algorithm updates are often fuzzy. It rarely offers actionable solutions or even targeted areas, such as content, usability, or links.

Thus diagnosing and fixing an organic search drop is much harder.

Google now broadly classifies algorithm updates as “core” or “helpful content.” Understanding the difference can make recovery easier and quicker.

Core Algorithm Update

A core algorithm update is Google’s reevaluation of an algorithm component. It doesn’t focus on specific sites or pages.

For example, a core update could target political bias in search results or a shift in query intent. It is a broad change to the algorithm, not a penalty. Per Google’s Search Central blog:

… there’s nothing in a core update that targets specific pages or sites. Instead, the changes are about improving how our systems assess content overall.

Core updates can address user experiences or on-page optimization tactics affecting the entire algorithm, not particular sites.

Hence sites don’t typically require action to recover from core updates. Sites can lose or gain traffic with each update without doing anything.

Yet sometimes entire sites are impacted — all pages lose traffic. That’s a good reason to assess your site’s overall content and usability for out-of-favor (to Google) tactics or elements.

Helpful Content

Helpful content is more of a penalty. It looks for content that targets search engines instead of humans and can penalize the whole site if most pages are presumed unhelpful.

Thus recovering from a helpful content update requires action. The next rollout won’t alter rankings without content changes sitewide.

Removing unhelpful content pages from your site may help. Regardless, a traffic drop coinciding with a helpful content update requires immediate action. There’s no recovery without it.

But a drop from a core update is more about analyzing organic listings and newly ranking URLs.

The helpful content algorithm is Google’s attempt to reward material that benefits humans. An update doesn’t elevate the rankings of superior pages; it penalizes unhelpful ones.

Core Update Helpful Content Update
Algorithm adjustment Modifier that targets specific signals
Doesn’t target specific on-page or on-site tactics Sitewide
Rewards Demotes
No fix required unless all pages affected Can be revoked by removing or improving unhelpful content

Commonalities

Both updates:

  • Are automated and machine-based. No human reviews or penalizes.
  • May impact the entire site. Recoveries occur only with the next update.
  • Are intent-based — to satisfy a searcher’s intent better. Intent analysis and optimization are key.

In short, while “core” and “helpful content” algorithm updates can focus on similar aspects, recovery from each is different. Helpful content addresses your site. Core updates address the algorithm.

Local business listings with Schema.org structured data

One of the things you can do to present your local business better in search results is Schema.org data for rich snippets. Adding structured data to your site can help search engines understand your business and how it performs. For this reason, you must add your NAP details, store/location map, reviews, and images. Find out how Schema.org structured data can help your local SEO.

Table of contents

Local customers use search engines

Modern-day customers use search engines to find your specific business and businesses around their current location. Customers using their phones looking for ‘Italian restaurants’ will get rich search results from local businesses. The results will include distance, reviews, opening hours, and possibly making a reservation.

Improve local rankings

Getting a good ranking for your local business means offering search engines as much data about your business as possible, which local SEO helps you do. Besides that, you need an excellent, fast, and mobile-friendly website, quality content, links, and reviews.

You should focus on being the most relevant result for a specific query to improve your rankings. In addition to that, your business has to be the best result. One way of getting this kind of recognition is by asking your customers for reviews. Reviews help search engines and potential customers determine which business is legit and which isn’t.

Why Schema.org

The main thing to remember is that Schema.org describes your data and content to search engines. Search engines can find out a lot about your site by crawling it. However, if you add structured data, you can give everything meaning. This way, search engines instantly grasp what the content means and how they should present it. In addition, Schema.org is a shared initiative by the big search engines, so using it will lead to consistent results in the respective search engines. Yoast SEO automatically adds structured data with much information about your site.

Rich search results for businesses

So, when we mean rich search results, we are talking about the information about a business you can directly see in search results without clicking a link. There are a few different results here: a regular organic search result for a business can feature breadcrumbs, highlighted pages, or even a search box. In Google, there’s also the Knowledge Panel on the right-hand side. Here, you’ll find lots of metadata about a business, from opening hours to photos. Finally, the results you see when you search for a specific term rather than a business. See the screenshots below for the different results for a particular or generic search.

Search for specific clothes store in Burbank
Looking for a local business using a specific term

Why you should use JSON-LD

To get rich results, you need to use structured data like Schema.org. In the past, adding Schema.org data to your post was pretty tricky because you had to embed it in your HTML code. With JSON-LD, you can add a JavaScript code block to your page. Plus, the code is readable and easy to change. Also, there are tons of helpful tools to do it for you.

With JSON-LD, you no longer have code wrapping around your HTML elements, with less possibility of messing things up. In addition to that, Google advises you to use it. Now, let’s see how it’s done.

How to add Schema.org to your local business listing

Since LocalBusiness is a subtype of Organization, we recommend implementing the fields for Organization in addition to the ones specific to your business.

When working on your listing, the most important thing to remember is to pick the correct business type. Make sure to choose a specific one, not a broad one. So, if you own a barbershop, you can use the LocalBusiness type HairSalon. You can find hundreds of types of businesses on Schema.org, so you’ll probably find one that matches closely. If not, try using the product types ontology. This site uses Wikipedia pages for describing products or services with GoodRelations and Schema.org. Here, you can get more specific information if your listing is too broad.

Generate and test

While you can write Schema.org JSON-LD code by hand, it’s not recommended. Use a generator like this Schema Markup Generator or Google’s Structured Data Helper — you can even ask ChatGPT to code structured data for you.

The Yoast Local SEO plugin is the easiest option to add proper local-oriented structured data. It also ensures that it plays nicely with the rest of the structured data generated by Yoast SEO.

Always validate your structured data in the Rich Results Test Tool. Using Schema Markup Validator, you can check your site to see if the structured data is implemented correctly. Or use Classy Schema to visualize your implementation.

Don’t forget to add your site to Search Console so you can check how Google presents your site. If you want to learn more about Google Search Console, read our beginner’s guide to Google Search Console.

Quickly add structured data for your local business

The Local SEO plugin by Yoast gives you everything you need to do well in the local search results pages!

Required properties for local businesses

There are two main Schema.org at play here: Schema.org/LocalBusiness and Schema.org/Organization. Here, you’ll find everything you need to inform search engines about your local business. To get started, you need to define at least the following properties: the name of your business, its postal address, a URL, and a logo. Here’s what a simple address looks like in structured data:

"address": {
  "@type": "PostalAddress",
  "streetAddress": "9901 Union Street",
  "addressLocality": "Simi Valley",
  "addressRegion": "CA",
  "postalCode": "93065",
  "addressCountry": "US"
}

The properties mentioned in the previous paragraph don’t get you far. You need to go further to make the most of structured data for your site. Be sure to add the following properties as well, if applicable. This is just the beginning, on Schema.org/LocalBusiness you’ll find loads more. Google also has a long list of supported properties.

- url (unlike the @id, should be a working link)
- geo
-- geo.latitude
-- geo.longitude 
- telephone
- aggregateRating
- openingHoursSpecification
- openingHoursSpecification.opens
- openingHoursSpecification.closes
- openingHoursSpecification.dayOfWeek
- openingHoursSpecification.validFrom
- openingHoursSpecification.validThrough
- menu
- department
- servesCuisine
- priceRange (how many $?)

Recommended properties for Organization

As LocalBusiness structured data is heavily tied to the Organization structured data, so it’s necessary to focus on that. There are loads of options to help Google understand your business better. Below is a list of recommended properties. Remember that Yoast SEO Premium and the Local SEO add-on make it easy to fill these in.

name (Text: name of your business)
alternateName (Text: other name you often use)
legalName (Text: the registered legal name)
description (Text: describe your business)
logo (URL or ImageObject)
url (URL)
sameAs (URL: links to other online services or social profiles)
telephone (Text)
email (Text)
address (PostalAddress)
 - address.streetAddress (Text)
 - address.addressLocality (Text)
 - address.addressRegion (Text)
 - address.postalCode (Text)
 - address.addressCountry (Text)
contactPoint (ContactPoint: how can a customer best contact you?)
 - contactPoint.telephone (Text)
 - contactPoint.email (Text)
numberOfEmployees (QuantitativeValue)
foundingDate (Date)
iso6523Code (Text)
duns (Text)
leiCode (Text)
naics (Text)
globalLocationNumber (Text)
vatID (Text)
taxID (Text)

Example code for local business Schema.org

To clarify how this works, we will use a real local business: Unique Vintage in Burbank, CA. This makes it a bit easier to validate the data we enter. In the code below, you’ll find all the NAP details, URLs, geolocation data, and opening hours you might need.


The LocalBusiness code above gets perfect scores from the Rich Results Testing Tool

Reviews

Reviews are a significant driver for new clientele. Scoring well in Google means your business provides quality, which can eventually lead to better local rankings. Think about how you pick the next company to visit. Will it be the one with three two-star reviews or the one with eighty-five-star reviews?

In the example above, we’ve added a review section. If you want to use reviews in your Schema.org data, remember that these reviews must live on your site. You cannot use sites like Yelp or TripAdvisor to generate reviews to show in the search engines. Ask your customers to leave a review. Make a review page, collect the reviews, and present them to the world.

Social

Another element to add to complete your online profile is links to your social media accounts. To do this, you must specify an organization or a person. The URL has to lead to your main site, while the sameAs links lead to your social media profiles.


Google My Business

There is another way you should add your local business to Google. By opening a Google My Business account, you will be able to verify that you are, in fact, the owner of your business. After that, you can add or edit all relevant information about your business, such as address information, opening hours, and photos. In addition to that, you can even manage the reviews people add to Google and see how your local listing performs.

Conversely, this only applies to Google. Every search engine can interpret Schema.org, so adding structured data to your site is still advisable. Additionally, Schema.org can do much more than add relevant local business locations. Therefore, Schema.org should be your main focus.

Despite all this, you’re still very much in Google’s hands. Some businesses appear in the Knowledge Panel, while others don’t. Some products get rich listings in the search results, including prices, reviews, and availability, while identical products from different vendors don’t. It’s hard to predict what will happen. However, don’t let this stop you.

Structured data for your local business

As we’ve shown, Schema.org can play an important part in the optimization of your site and your SEO strategy. Structured data can do much more; look at all those properties on Schema.org. We’ll keep an eye on what structured data can do for your site and keep you in the loop!

Don’t forget that if you want an easier way to add your local business and organization data to your pages, you should check out our Local SEO plugin and Yoast SEO Premium.

Read more: Structured data with Schema.org: the ultimate guide »

Coming up next!

How Google Uses Keywords

Years ago Google ranked organic listings by exactly matching search queries to text on a web page. Search engine optimization involved creating multiple pages containing all sorts of keyword variations.

The result was an avalanche of inferior content ranking at or near the top for long-tail queries.

Search engines no longer rely on exact-match keywords, instead attempting to understand the intent and context behind a query. It’s common for Google to rank a page that doesn’t contain a single query word.

For example, a Google search for “affordable airfare” produces page-1 organic results mainly containing “cheap flights,” “last minute flight deals,” and similar. None that I saw included the exact phrase “affordable airfare.”

And searching “trails for kids upstate ny” generates organic results for “family-friendly hikes in Albany and “easy hikes near NYC” — both likely satisfying the searcher’s intent.

Searching “trails for kids upstate ny” generates results for “family-friendly hikes in Albany and “easy hikes near NYC.”

Google’s “How search works” portal explains the process:

To return relevant results, we first need to establish what you’re looking for — the intent behind your query. To do this, we build language models to try to decipher how the relatively few words you enter into the search box match up to the most useful content available.

This involves steps as seemingly simple as recognizing and correcting spelling mistakes and extends to trying to our sophisticated synonym system that allows us to find relevant documents even if they don’t contain the exact words you used.

Keyword Research

Is keyword research important for SEO in 2024?

While it no longer relies on exact-match queries, Google’s algorithm still uses keywords. In the same “How search works” page, Google states:

The most basic signal that information is relevant is when content contains the same keywords as your search query. For example, with webpages, if those keywords appear on the page, or if they appear in the headings or body of the text, the information might be more relevant.

Further down Google says it also uses “other relevancy signals” (presumably beyond keywords), “interaction data to assess whether search results are relevant to queries,” and machine learning to make sense of the data.

Google provides an example:

Just think: when you search for “dogs,” you likely don’t want a page with the word “dogs” on it hundreds of times. With that in mind, algorithms assess if a page contains other relevant content beyond the keyword “dogs” — such as pictures of dogs, videos, or even a list of breeds.

Yet the benefits of keyword research extend beyond SEO.

Keywords:

  • Help understand the needs of target customers.
  • Guage demand — high search volume signals higher demand for a product or service.
  • Guide site structure — keywords with heavy search volume could be categories; lower volume subcategories.
  • Identify potential high-traffic content ideas.
  • Inform new products or categories.

Keyword Optimization

Optimizing for exact-match keywords is not just dated. It may be counterproductive owing to Google’s Helpful Content algorithm, which devalues websites targeting search engines rather than humans.

Instead:

  • Search Google for topics and formats it considers helpful to searchers.
  • Prompt ChatGPT to create personas of searchers using your keywords, including their intent and likely needs. Google uses AI similarly to identify which content better serves searchers.
  • Identify related keywords for products that solve similar problems.
  • Brainstorm content elements for a keyword that improve engagement, such as a table or checklist.

In short, keywords remain the “most basic signal that information is relevant” to the query, per Google. Use a primary keyword in a page’s title and meta description, but don’t force it.