Apr 18 2024
Extending Your Schema Markup From Rich Results To A Knowledge Graph via @sejournal, @marthavanberkel

You probably know Schema Markup for its ability to help pages achieve a rich result. But did you also know that you can use Schema Markup to build a reusable content knowledge graph for your organization’s web content?

When SEO professionals do Schema Markup with the sole intention of achieving a rich result, they miss the opportunity to create semantic Schema Markup and build their content knowledge graph.

Content knowledge graphs enable search engines to contextualize the content on your site and make inferences more easily.

Let me illustrate the power of Schema Markup for inferencing by introducing myself. My name is Martha van Berkel, and this is my knowledge graph. It explains who I am and how I relate to other things.

  • I studied at MIT and have a degree in Mathematics and Engineering.
  • I am Canadian, the co-founder and CEO of Schema App, and I know a lot about Schema Markup.
  • I worked at Cisco for 14 years.
  • I also used to own a 1965 Austin Healey Sprite, and my car was in the movie “Losing Chase,” which Kevin Bacon directed. In fact, Kevin Bacon used to drive my car.
Martha van Berkel's knowledge graphImage created by author, March 2024

What can you infer from my knowledge graph? Are you thinking about how you can win the “6 degrees of separation from Kevin Bacon” game? Or are you thinking about how it makes sense that I know about Schema Markup and am writing about it because I am the CEO of Schema App?

You can make these inferences because of the understanding you’ve developed from reading the specific relationships within my knowledge graph. I used the properties for a Person Schema.org type to describe and define my relationship with these things, the same language we use to optimize our web pages.

If I didn’t specifically define the relationship between me and these other things, you might think that I work at Cisco, am a user of Schema App, and am related to Kevin Bacon. This is why being specific and adding context is important!

So, just like I used Schema Markup to bring context and clarity to who I am, you can use Schema Markup to add more context to your website content so that search engines and AI can make accurate, powerful inferences about it.

In this article, we will discuss why it is important to start thinking about Schema Markup for its semantic value and how you can use your Schema Markup to build a reusable content knowledge graph.

Why Is It Important To Start Thinking About Schema Markup For Its Semantic Value?

The search landscape is evolving quickly. Search engines are racing to provide a new search experience that leverages inferencing and chat experiences.

We see this in Google’s Gemini and Bing’s ChatGPT and from new entrants, such as Perplexity AI. In the chat experience, search engines need to be able to provide users with quick, accurate answers and deal with evolving contexts.

Consumers are now also using more hyper-longtail queries in their searches. Instead of searching for [female doctor Nashville women’s health], they are searching for [find me a female doctor who can help me with my cramps and has an appointment available within the next 2 days].

Search engines and large language models (LLMs) cannot easily infer the answer to this query across a company’s website data without understanding how the information is connected. This contextual inference is why search engines have moved from lexical to semantic search.

So, how do you make it easy for these machines to understand and infer things from your content? By translating your web content into a standardized vocabulary understood by humans and search engines – Schema Markup.

When you implement Schema Markup, you can identify and describe the things, also known as entities, on your site and use the schema.org properties to explain how they are related.

Entities are unique, well-defined, and distinguishable things or concepts. An entity can be a person, a place, or even a concept, and it has attributes and characteristics.

Your website content discusses entities related to your organization (e.g., brand, products, services, people, locations, etc.), and you can use Schema Markup to describe your entities and connect them with other entities on your site.

Entities are the foundational building blocks of a content knowledge graph.

The value of a content knowledge graph far exceeds SEO. Gartner’s 2024 Emerging Tech Impact Radar report identified knowledge graphs as a key software enabler and important investment to enable generative AI adoption.

Many AI projects are powered by large language models prone to hallucination and errors. Research shows that when paired together, knowledge graphs can provide factual knowledge, resulting in more accurate answers from the LLM.

By creating a content knowledge graph through Schema Markup, SEO pros can enable search engine understanding and help prepare their organization to be leaders in innovations with AI.

Read More: Entities & Ontologies: The Future Of SEO?

Implementing Schema Markup To Build A Content Knowledge Graph Vs. Just Rich Results

You might wonder: How is this different from implementing Schema Markup to achieve a rich result?

When an SEO pro’s goal is to achieve a rich result, they tend to only add Schema Markup to the pages and content that are eligible for the rich results. As such, they are only telling search engines small parts of the organization’s story.

They are not providing search engines with detailed information or context about the entities on their site and how they are connected to one another.

This leaves search engines to guess the intentions and meaning of their content – just like you might think I’m related to Kevin Bacon and work at Cisco if I didn’t establish my relationship with these things in my introduction.

When an SEO pro’s goal is to build a content knowledge graph, they use Schema Markup to identify, describe, and explain the relationship between the entities on their site so that search engines can truly understand and contextualize the information on their organization’s website.

So, how do you start creating your Schema Markup with the intention of building a knowledge graph?

How To Implement Schema Markup To Build Your Content Knowledge Graph

1. Identify The Pages On Your Website That Describe Your Key Entities

Your website can contain thousands of entities (like specific products, individuals, services, locations, and more).

However, certain entities are important to your business goals and outcomes. This content is often what you need people and search engines to know about so that you can convert them into customers or inform them about your brand.

Common key entities often include your organization, services, products, people, and brand, but this ultimately depends on your business objectives.

For example, if you are a healthcare provider looking to establish a trusted reputation and drive appointment bookings through your website, your key entities could include your organization, medical facilities, physicians, and services offered.

Once you’ve identified which entities are important to your organization, you can find the page on your site that best represents them. Ideally, each page would define one entity and how it relates to other entities on the site.

2. Use The Schema.org Vocabulary To Describe The Entities

When you implement Schema Markup on a page, you are using the Schema.org vocabulary to make a series of statements that describe the entity. The Schema.org type categorizes the entity, while the Schema.org property describes the entity.

For example, a physician detail page might include information about the physician’s name, medical specialty, who they work for, the hospital or medical clinic they work at, the medical services they provide, and the geographical area they serve.

You can use Schema Markup to describe these aspects of the entity and express it as a graph with specific connections.

Schema exampleImage from author, March 2024

This helps search engines understand details about the physician to provide answers to a detailed query like [find me a cardiologist near me who can perform an EKG and has an appointment available in the next 2 days].

Every page on your website describes something about your business.

Implementing Schema Markup on each page clearly tells search engines what the page is about and how its concepts relate to other concepts on your website. Now, search engines and large language models can use this data to make inferences and confidently answer specific queries.

3. Connect The Entities On Your Website

Even though each web page is home to a unique entity, the content on your webpage might mention other entities that you’ve defined on other pages of your site.

If you want to build a content knowledge graph, you have to showcase how the entities on your website are connected and provide context using the right schema.org property.

This goes beyond a hyperlink connecting both pages using anchor text. With Schema Markup, you use the Schema.org properties that best describe the relationship to connect the entities.

For example, if the Physician works for the organization HealthNetwork, we can use the memberOf property to state that the Physician is a memberOf the Organization HealthNetwork.

Example of Physician Organization relationship using Schema MarkupImage from author, March 2024

When you look at the content on the page, if there are URLs linked to take you to another step in the journey, these are entities that should be linked within the Schema Markup. For physicians, this may be the service line pages, the hospitals where they practice, etc.

This provides search engines with more contextual information about the physician, which enables them to answer more complex queries.

Using these basics, you have started building your content knowledge graph. This should be done in addition to trying to achieve rich results. However, the properties you use to connect your entities are likely different from Google’s required properties for the rich result.

4. Link Your Entities To Other External Authoritative Knowledge Bases To Disambiguate Them

In addition to connecting the entities on your site, you can further define the entities mentioned on your pages by linking them to known entities on external authoritative knowledge bases like Wikipedia, Wikidata, and Google’s Knowledge Graph.

This is known as entity linking.

Entity linking can help you define the entities mentioned in your text more explicitly so that search engines can disambiguate the entity identified on your site with greater confidence and showcase your page for more relevant queries.

At Schema App, we’ve tested how entity linking can impact SEO. We found that disambiguating entities like places resulted in pages performing better on [near me] and other location-based search queries.

Our experiments also showed that entity linking can help pages show up for more relevant non-branded search queries, increasing click-through rates to the pages.

Here’s an example of entity linking. If your page talks about “Paris”, it can be confusing to search engines because there are several cities in the world named Paris.

If you are talking about the city of Paris in Ontario, Canada, you can use the sameAs property to link the Paris entity on your site to the known Paris, Ontario entity on Wikipedia, Wikidata, and Google’s Knowledge Graph.

Example of Entity LinkingImage from author, March 2024

Content Knowledge Graph Brings Context To Your Content

If your organization is using Schema Markup on select pages for the purpose of achieving a rich result, it is time to rethink your Schema Markup strategy.

Rich results can come and go.

However, the content knowledge graph you create using your Schema Markup can help search engines better understand and infer things about your organization through your content and prepare your organization to innovate with AI.

Like it or not, knowledge graphs are here to stay and you can start building yours by implementing proper semantic Schema Markup on your site.

More resources:

Featured Image: Gracia Chua/Schema App

Ecommerce MGMT 0 Comments
Apr 16 2024
Google Clarifies Vacation Rental Structured Data via @sejournal, @martinibuster

Google’s structured data documentation for vacation rentals was recently updated to require more specific data in a change that is more of a clarification than it is a change in requirements. This change was made without any formal announcement or notation in the developer pages changelog.

Vacation Rentals Structured Data

These specific structured data types makes vacation rental information eligible for rich results that are specific to these kinds of rentals. However it’s not available to all websites. Vacation rental owners are required to be connected to a Google Technical Account Manager and have access to the Google Hotel Center platform.

VacationRental Structured Data Type Definitions

The primary changes were made to the structured data property type definitions where Google defines what the required and recommended property types are.

The changes to the documentation is in the section governing the Recommended properties and represents a clarification of the recommendations rather than a change in what Google requires.

The primary changes were made to the structured data type definitions where Google defines what the required and recommended property types are.

The changes to the documentation is in the section governing the Recommended properties and represents a clarification of the recommendations rather than a change in what Google requires.

Address Schema.org property

This is a subtle change but it’s important because it now represents a recommendation that requires more precise data.

This is what was recommended before:

“streetAddress”: “1600 Amphitheatre Pkwy.”

This is what it now recommends:

“streetAddress”: “1600 Amphitheatre Pkwy, Unit 6E”

Address Property Change Description

The most substantial change is to the description of what the “address” property is, becoming more descriptive and precise about what is recommended.

The description before the change:

PostalAddress
Information about the street address of the listing. Include all properties that apply to your country.

The description after the change:

PostalAddress
The full, physical location of the vacation rental.
Provide the street address, city, state or region, and postal code for the vacation rental. If applicable, provide the unit or apartment number.
Note that P.O. boxes or other mailing-only addresses are not considered full, physical addresses.

This is repeated in the section for address.streetAddress property

This is what it recommended before:

address.streetAddress Text
The full street address of your vacation listing.

And this is what it recommends now:

address.streetAddress Text
The full street address of your vacation listing, including the unit or apartment number if applicable.

Clarification And Not A Change

Although these updates don’t represent a change in Google’s guidance they are nonetheless important because they offer clearer guidance with less ambiguity as to what is recommended.

Read the updated structured data guidance:

Vacation rental (VacationRental) structured data

Featured Image by Shutterstock/New Africa

Ecommerce MGMT 0 Comments
Apr 12 2024
SEO Takeaways from SGE’s Partial Rollout

Google extended Search Generative Experience last month beyond Labs, its testing program. A limited number of searchers now see AI snapshots in results regardless of whether they signed up.

Many observers believe it’s the first step to SGE becoming fully public this year. Google hasn’t much changed SGE in Labs for months, perhaps signaling its satisfaction thus far.

I’ve closely monitored SGE developments. Here are my observations and expectations.

Traffic Losses Overestimated

Authoritas, a search-engine-optimization platform, has been testing SGE in Labs and publishing the results. In March, the tests found the SGE appears in 91% of U.S. search results for brand and product terms in two ways.

First, the results could contain a “Generate” button that produces an AI-powered answer only when clicked, such as this query of “best laptops.”

Clicking the “Generate” button produces AI-powered answers. Click image to enlarge.

Second, the search results could contain an instant answer, such as the example below for “How healthy is spaghetti squash?” Clicking “Show more” expands the explanation.

Instant SGE answers such as this example appear automatically. Clicking “Show more” expands the explanation. Click image to enlarge.

The instant answer takes much more SERP space and will likely steal more clicks from organic listings because it pushes them further down the page.

Fortunately, according to the same Authoritas study, clicks on the “Generate” button drive 81.4% of SGE responses, much more than an instant answer. This indicates that organic listings won’t be hugely impacted, at least for now, since there’s no page disruption unless the button is clicked.

Organic Listings Displaced

However, when SGE is triggered, organic results appear far below the initial screen.

For example, searching for “smart TV” and clicking “Generate” produces an AI answer that occupies an entire screen on mobile and desktop.

Authoritas estimated an average organic listings drop of 1,243 pixels, depending on the search term. SGE results more or less eliminate the visibility of organic listings, especially for queries seeking consumables such as household goods.

Even before SGE, organic visibility was increasingly limited owing to the various features Google inserts at or near the top, such as ads, “People also ask” boxes, image packs, local packs, and more.

Opportunities in SGE

The good news is that SGE answers contain links, an opportunity for organic visibility. Authoritas states that SGE’s snapshots, on average, contain five unique links, and just one matches the top 10 organic listings below it. Perhaps it’s because the snapshots often address related queries.

For example, SGE snapshots for the search of “best laptops” list many makes and models as well as links for the best laptops for students, budgets, and coding. Organic listings for “best laptops” do not include those additional choices.

SGE snapshots for “best laptops” list many makes and models as well as related links, such as “The Best Laptops for Sims 4” (for students). Click image to enlarge.

Thus after optimizing important keywords, consider creating supporting content for related queries, increasing the chances of showing up in SGE results. For ideas, keep an eye on “Related searches” because those keywords seem to appear in SGE.

“Related searches” keywords seem to appear in SGE.

Ecommerce MGMT 0 Comments
seo enhancements
Apr 10 2024
How do you improve your mobile site?

Your site should be mobile-friendly. Because nowadays, most people are searching on Google with their phone. But what does it mean to have a mobile-friendly site? And where do you start? In this SEO basics article, you’ll find an overview of what you could do to improve your mobile site.

Table of contents

When is a site mobile-friendly?

A site is mobile-friendly when it:

  • helps users get their tasks done quickly and joyfully.
  • loads correctly on a mobile device like a smartphone or tablet.
  • loads lightning-fast.
  • presents content in a readable fashion, without users having to pinch and zoom.
  • offers enough space to navigate by touch.
  • offers added value for mobile users.
  • is instantly understandable for search engines.

Why is mobile SEO important?

Mobile SEO makes sure your mobile site offers the best possible presentation of your content to a mobile device user. Since our world is increasingly mobile-oriented, it’s important that your site is mobile-friendly. If your site isn’t (properly) available for mobile users, you’ll miss out on a decent ranking in the search engines, and thus miss the income. Therefore, you should do everything in your power to optimize the mobile view of your site and make it as good as possible. In fact, it should be excellent!

Important to Google

Since 2016, Google uses the mobile version of a site to determine the site’s rankings. So if your site isn’t up to scratch or shows less content on your mobile site, you’ll find it difficult to get good rankings. That’s why it’s so important to create a fully functioning and responsive design for your (mobile) site.

Luckily, Google has a great getting started guide to help you improve your mobile site. Plus, they’ve also set up a Page Experience initiative that gives you metrics — the so-called Core Web Vitals — on how humans and machines perceive your site’s performance. So, use these metrics to help you figure out what to focus on while improving your mobile site.

Treat it as one website

Don’t forget to see your site as being a single thing. You shouldn’t have a ‘mobile site’ that’s distinct from your ‘desktop site.’ You should have one site that adapts to whatever screen it’s being viewed on. That also means that the content of the different views should be the same.

How to improve your mobile website

To improve your mobile SEO, you need to focus on a couple of things:

  • Make a joyful user experience.
  • Make sure your site is responsive.
  • Improve your site speed.
  • Use structured data.
  • Don’t block JavaScript, HTML and CSS code.
  • Don’t use too many redirects.
  • Choose the correct viewport.
  • Don’t use interstitials or pop-ups.
  • Verify mobile-friendliness.
  • Tell Google about your site.

Let’s go over these topics in more detail.

Focus on making your site easy and joyful to use with mobile SEO

Offer a great user experience to your users, and you’ll notice that Google will enjoy it too. So, how do you do that? First, figure out what you want users to do on your site. Then, make sure that it’s easy for people to do. Do you want people to call you? Make sure you put your phone number front and center, so it’s easy to find. Want to enhance conversions? Make that buy button stand out and function properly! In other words: bring focus to your site, and helpfully guide your visitors through the steps you want them to take.

But don’t just focus on your intent. Look at your users too! Figure out why they visit and which tasks they mostly do on your site. Then make sure it’s easy for them. Because If something frustrates your user, it hurts you and your results. That’s why you should test, improve, and fully optimize your mobile site.

Responsive design

There are multiple ways to improve your site so it’s available for mobile users. The most important one is to create a responsive design. This is also the the technology that Google advocates. With a responsive design, your site lives on one URL, which makes it easier for Google to understand and index it.

If you use WordPress, chances are your theme is already responsive and can adapt to all screens. Still, it’s good to check how your site scales in Google Chrome’s Developer Tools. Because if it doesn’t scale correctly, you should talk to your web developer about fixing it – or choose a different theme.

Improve your site speed

One of the most important things you can do to improve your site’s mobile SEO is to improve the site’s loading speed. Time after time, studies have shown that people leave sites that load slowly, and probably never return. That’s why site speed has been a ranking factor for years, and why Google is increasingly focusing on fixing this common issue. See the Page Experience update and the Core Web Vitals metrics’ introduction for more proof.

If you need more tips, we have a post on how to improve your site speed and which tools that might help you.

Get better web hosting for your site

The number one tip to optimize the speed of your mobile site is to invest in better web hosting. Many sites run on budget hosts that share a lot of the server space with other websites, which can cause their sites to slow down. That’s why it really is essential to stay away from cheap hosting and get a good plan at a renowned host — it truly pays for itself!

Don’t know where to start? We have a page with WordPress web hosting companies that we vouch for, as we vetted them personally.

Optimize images

If there is one quick win to improve your site speed, it’s this: optimize your images. Don’t load those 3000 x 2000 pixel HD images on your site. Scale them to the correct size, then make them smaller with tools like ImageOptim, Squoosh, or WordPress plugins like WP Smush. You can also look into serving those images in next-gen image formats like WebP.

Minify code

Every request your site has to make has an impact on your site speed. That’s why you have to work on reducing these requests to improve your mobile site. One way to do this is by minifying code.

Minifying code means that you group and link together assets like JavaScript and CSS. As a result, the browser has to load fewer files, which leads to a faster site. This sounds hard to implement, but a plugin like WP Rocket can take care of all your caching needs. Or you can use Cloudflare’s Automatic Platform Optimization for WordPress to get a load of enhancements in one go.

Browser caching

By using browser caching, you’re telling the browser that page elements that don’t change often can be saved inside its cache. This way, the browser only has to download new and dynamic content whenever it visits again. Again, this is something that a plugin like WP Rocket can help you with. Or you can also do it yourself if you like.

Reduce redirects

A redirect leads a visitor from one requested page to another, because the requested page was moved or deleted. While this leads to a good user experience if done well, the more redirects you use, the slower your site will be. Don’t make endless redirects. Also, try not to keep links around that point to deleted posts redirected to new ones. Always make direct links.

Use structured data to improve your mobile site

Structured data is essential for every site. With structured data, you can describe your content in a way that search engines can understand. It gives you a direct line of communication with search engines, so to say. In return, search engines might reward you with awesome rich results.

Your mobile site needs to have the same structured data as your desktop variant — otherwise, Google might get confused. Yoast SEO automatically adds structured data for the most important parts of your site, which you can fine-tune it to your liking.

Don’t block assets like JavaScript, HTML and CSS

We’ve said it before, and we’re going to keep saying it: Don’t block assets like JavaScript, HTML and CSS. Doing so makes it harder for Google to access your site and that could lead to bad rankings. Check your Google Search Console to see if you’re blocking resources. If so, we advise that you take away all blockades if you want to truly optimize your mobile site.

Improve legibility

Make sure that your mobile site is readable on mobile devices. Use different devices to check if your typography is in order and make changes when necessary. Typography can make or break the user experience of your site.

Improve tap target sizes

People hate it when their fingers can’t hit a button, link, or menu item without fault. They can feel frustrated when navigation is hard or unnatural. Please fix it to improve your mobile site.

Choose the correct viewport

The viewport determines the width of the page for the device used to view it. By specifying a correct viewport, you make sure that visitors with specific devices get the right version of your site. Fail to do this, and you might show your desktop site to a small-screen smartphone user — a big no-no.

Don’t use interstitials or pop-ups

Google will penalize sites that use large pop-ups or interstitials to promote newsletters, sign-up forms, or ads. These often get in the way of the user quickly accessing the content they requested. Don’t use these. If you must though, make sure you abide by Google’s rules.

Test your site and tell Google about it

Before you start working on your mobile SEO, you should run a mobile usability test on Google to see where you should start. As you work, you should keep testing to see if you’re making progress. If your mobile site is optimized, you need to tell Google so your site will be checked and indexed. Use Search Console to stay on top of the performance of your site.

Investigate other technologies

There are other ways to improve the performance of your mobile site. One of these technologies is the Accelerated Mobile Pages (AMP) framework. This is an initiative by Google and others to get web pages to load super fast on mobile devices. By wrapping your content in special HTML code, you can optimize the pages in a way that Google can use to improve the performance. Keep in mind that AMP is not without its drawbacks, and not every project will benefit from it.

AMP is not the only technology that helps you optimize your mobile site. Other companies offer similar solutions, like Cloudflare’s various optimized delivery technologies. There are so many options these days!

Conclusion

Mobile is the new baseline, the new default. Do everything you can to fix your mobile site and make it perfect, not just in Google’s eyes, but, more importantly, your visitors. Mobile SEO is not just about great content and a flawless technical presentation. It’s more about creating an excellent user experience. Once you’ve achieved that, you’re on your way to the top!

Read more: Mobile SEO: the ultimate guide »



Edwin Toonen

Edwin is a strategic content specialist. Before joining Yoast, he spent years honing his skill at The Netherlands’ leading web design magazine.

Avatar of Edwin Toonen

Coming up next!

Ecommerce MGMT 0 Comments
Mar 26 2024
All about Core Web Vitals: INP (Interaction to Next Paint)

Google’s Core Web Vitals have emerged as critical metrics for SEO. These metrics help you optimize your websites for a superior user experience. A new player is making headlines among these vital metrics: Interaction to Next Paint (INP). This one replaces the First Input Delay (FID). This post will explain what INP entails, its significance, and how to improve your site’s performance for SEO.

Essence of Interaction to Next Paint (INP)

Interaction to Next Paint measures the responsiveness of a web page to user inputs, such as clicks, taps, and keypresses. It represents the time from when a user interacts with your page to when they see a response on the screen. Unlike its predecessor, First Input Delay, which only accounted for the first input, INP provides a broader view by capturing the responsiveness throughout the life of the page.

Google is very dedicated to enhance the user experiences offered by sites. To validate these, it introduces a more nuanced and comprehensive metrics. It now introduces Interaction to Next Paint into the Core Web Vitals. INP measures a critical aspect of the user’s experience — the responsiveness of a page to user interaction.

By integrating INP into Core Web Vitals, Google aims to provide developers with a complete picture of their page’s performance. In addition, it encourages improvements that genuinely enhance the user experience.

Why INP matters

A seamless user experience is the cornerstone of successful SEO. Interaction to Next Paint directly influences how users perceive the efficiency and usability of a webpage. Pages that respond swiftly to user interactions are more likely to engage visitors. Better performance can reduce bounce rates, and, ultimately, higher rankings in search results.

As the transition from FID to INP unfolds, webmasters and SEO experts must embrace this broader metric. Understanding and optimizing for INP will be crucial for maintaining and improving search rankings.

Real-world improvements for yoast.com

Despite the challenges in optimizing for INP, our team at Yoast has remarkably improved responsiveness. By focusing on efficient code execution and minimizing render-blocking resources, we have significantly enhanced our site’s performance.

Google Search Console already provides INP reports, splitting into mobile and desktop issues. At Yoast, we’ve used these to guide our optimizations. In addition, Screaming Frog now includes INP pass/fail within their crawl reports, which helps as well.

Below shows how the work we did in December and January has reduced the number of issues dramatically:

INP score on desktop
INP score on mobile

But remember, while it’s always great to have zero errors, don’t obsess about cutting off milliseconds to get there. If there are significant performance issues, then solve these as soon as you can. Always keep in mind, though, don’t spend dollars to save pennies! Focus on the general page experience; things will naturally progress from there.

Improving Interaction to Next Paint

The shift to INP necessitates a fresh approach to measuring and enhancing web performance. Tools like Google’s Lighthouse, PageSpeed Insights, and the Chrome User Experience Report offer valuable insights into INP scores and opportunities for optimization.

Practical strategies to enhance your INP score

Improving your Interaction to Next Paint INP score benefits your site’s user experience. It’s an important part of staying competitive in SEO. Here are actionable tips to help you enhance your INP score:

1. Optimize event callbacks

Event callbacks are at the heart of user interactions. Reducing the time these callbacks take to process can significantly improve your INP score. Assess the complexity of your event handlers and streamline their code to ensure quick execution.

2. Avoid blocking the main thread

The main thread is where the browser processes user events, executes JavaScript, and renders updates to the screen. Keeping it unblocked ensures that the page can respond to user inputs promptly. Avoid heavy computations or long-running tasks on the main thread to prevent delays in responsiveness.

3. Break up long tasks

Tasks taking more than 50 milliseconds can interfere with the page’s ability to respond to user inputs effectively. Breaking these long tasks into smaller chunks allows the browser to intersperse input handling between these tasks, improving the overall responsiveness.

4. Optimize JavaScript execution

JavaScript can significantly impact your page’s responsiveness. Optimizing how JavaScript is loaded and executed on your page can improve INP scores. Techniques include deferring non-critical JavaScript, using async scripts, and removing unused code.

5. Minimize unnecessary tasks

Evaluate the tasks on your page and identify any that are not essential to the immediate user experience. Postponing or eliminating unnecessary tasks can free up resources, allowing the browser to prioritize user interactions.

6. Prioritize important actions

Not all tasks are created equal. By prioritizing important actions — such as those directly related to user interactions — you ensure that these tasks are executed first, leading to a smoother and more responsive experience.

7. Leverage requestIdleCallback

The requestIdleCallback API allows you to schedule background tasks to run when the browser is idle. This is particularly useful for tasks not critical to the immediate user experience. By using requestIdleCallback, you ensure these tasks do not interfere with the page’s responsiveness to user inputs.

Continuous improvements

Implementing these strategies requires a thoughtful approach to web development and an understanding of how user interactions are processed. Tools like Lighthouse and PageSpeed Insights can provide insights into your Interaction to Next Paint score. In addition, these can identify specific areas for improvement.

You can significantly enhance your site’s responsiveness by optimizing event callbacks, minimizing main thread blockage, breaking up long tasks, and prioritizing user-centric actions. This leads to a better user experience. It also aligns with Google’s emphasis on page responsiveness as a critical SEO component in the Core Web Vitals era.

Improving INP is a continuous process that can lead to substantial gains in user satisfaction and engagement. As you implement these changes, monitor your site’s performance. Check the impact on your INP scores and refine your strategies for even better results.

Looking ahead

The introduction of INP signals Google’s ongoing commitment to refining its page experience signals. Staying informed and proactive in optimizing for INP and other Core Web Vitals is imperative for you to excel in SEO.

Interaction to Next Paint is a pivotal metric for assessing and enhancing web page responsiveness. Understand its nuances, embrace the available tools, and implement data-driven optimization strategies. Esure that your website meets the ever-changing standards of user experience and SEO.

Let’s continue the conversation in the comments below. Share your experiences, challenges, and successes in working to improve INP. Together, let’s prepare those sites for lift-off!



Edwin Toonen

Edwin is a strategic content specialist. Before joining Yoast, he spent years honing his skill at The Netherlands’ leading web design magazine.

Avatar of Edwin Toonen

Coming up next!

Ecommerce MGMT 0 Comments
Mar 22 2024
Traffic Recovery from Domain Changes

Businesses change website domains for many reasons — rebranding, acquisition, product changes, and more. While it is often necessary, a new domain reduces organic search traffic by upwards of 30% in my experience.

Similar to a new URL structure, domain changes require 301 redirects, which leak equity. I’ve seen rankings slip for months afterward. If the decline persists, a detailed audit could help.

Here are my go-to tools for domain-change audits.

Google Search Console

The first step is to keep the abandoned domain verified within Search Console to access historical ranking and click data.

Next, after verifying the ownership of your new domain, notify Google of the switch via Search Console’s “Change of Address” tool at Settings > Change of address.

Notify Google of the new domain via Search Console’s “Change of Address” tool. Click image to enlarge.

Search Console can identify which pages or folders lost the most traffic from the move. For this, open the Performance > Pages reports for the old and new domains (each in separate tabs). By default, both URL lists are sorted by the number of organic clicks, making it easy to compare the top-performing pages.

Don’t delay the research, as Search Console shows only 16 months of data. I typically access the report two, three, and four months afterward to monitor shifts.

It’s a good sign if no sections, categories, or URLs decline more than others — the move likely produced no structural problems.

If some areas slipped more than others, turn to the Wayback Machine.

Wayback Machine

Domain changes frequently include altering the site structure, especially with acquisitions or product differences. Wayback Machine helps identify which structural changes lowered traffic.

Having noted in Search Console the URLs with material drops, compare archived versions of those pages in Wayback Machine to the new ones, ensuring:

  • The main navigation includes the URLs as prominently as the previous domain,
  • The URLs are interlinked throughout the site — in related products, guides, sidebar widgets, footers, and so on.

Semrush

A loss in link equity from the old site to the new could account for the traffic decline. Semrush can help pinpoint the loss.

To start, run both domains through the tool’s primary analysis bar. Then proceed to Backlink Analytics > Indexed Pages and check the box “Broken links.” This report for both sites will list the pages that have broken backlinks and return 404 or 410 errors.

Semrush’s “Broken links” report lists the pages that return 404 or 410 errors. Click image to enlarge.

Screaming Frog

Domain changes often produce unexpected structural glitches. Hence it’s critical to run the new site through a web crawler such as Screaming Frog, JetOctopus, or similar.

Most crawlers allow limited access for free. Screaming Frog, for example, crawls 500 pages without charge. Beyond that, the paid version is $259 per year. It includes a crawl-compare feature, handy for contrasting crawls of the old site and the new.

When crawling a site, look for redirect chains. These trap external and internal link equity and extend the crawl time. Both may cause traffic declines.

Ecommerce MGMT 0 Comments
Mar 20 2024
2024 SEO Tools & Tactics To Refresh Your Strategy & Boost Performance via @sejournal, @ahrefs

This post was sponsored by Ahrefs. The opinions expressed in this article are the sponsor’s own.

With AI taking the digital world over, constant Google algorithm changes, and growing competition, how can you shift your strategy to keep up?

Constant changes mean SEO professionals need to find new ways to reach SEO success in 2024.

If you’re looking to not simply survive but thrive amidst these seismic changes, it’s important to keep up with the latest trends and look for new ways to work on SEO.

In this guide, we’ll uncover strategies to help you increase search discoverability, and we’ll introduce the tools you’ll need to implement them effectively.

Some of the most significant improvements you can start making in 2024 include:

  • Knowing how to optimize processes with AI.
  • Mastering technical SEO and website optimization.
  • Executing and measuring your SEO more efficiently.

How AI Capabilities Can Be Applied To SEO

Everyone’s talking about how to best harness the power of Artificial Intelligence (AI) in 2024.


But if there’s one thing you should take away from the AI conversation, it’s this: AI can’t replace humans.

Utilizing AI in 2024 is about striking the perfect balance and finding the synergy between human ingenuity and AI advancements.

Rather than relying solely on AI technology, find ways to use it to enhance your results.

For instance, if you’re a content marketer looking to streamline your process, AI could assist by offering insights, recommendations, and data-driven strategies to elevate the quality and relevance of your content.

How Ahrefs Integrates AI in SEO & Content Marketing Tools

With modern tools, keyword research became a simple and streamlined task. Except for finding good seed keywords to start with.

If you’re familiar with Ahrefs’ Keywords Explorer tool, you’ll be pleased to know that it was improved with a new AI assistant that effortlessly generates keyword suggestions.

You can get AI keyword suggestions directly within the platform, without needing to go back and forth with ChatGPT when doing your keyword research.

And when it comes to content marketing applications for AI, Ahrefs released a number of free AI-powered writing tools. With these tools you can:

  • Input your rough ideas and get an organized, well-structured outline in minutes.
  • Improve the quality, clarity, and readability of a sentence or paragraph with an instant content refresh.
  • Generate optimized meta titles for better search engine visibility.
  • Craft informative, SEO-friendly meta descriptions for your articles quickly and easily.
  • Simplify and summarize your content with precision.
  • Brainstorm variations of ready-to-use, SEO-friendly blog post ideas to drive more traffic to your blog.
  • Generate descriptive alt text for your images to improve accessibility and SEO without a hassle.
  • Get inspiration for your next piece of content by generating a variety of creative ideas.
  • And more!

Stay ahead of the curve and start leveraging AI to your advantage with Ahrefs.

How To Master Website Optimization & Technical SEO

A well-optimized and technically sound website acts as a sturdy foundation, insulating you from the impact of ruthless core updates that search engines may roll out.


With search algorithms becoming increasingly sophisticated, seamless user experience, fast loading times, and adherence to Core Web Vitals have become critical factors in SEO success.

By mastering technical SEO and keeping your site in top-notch condition, you’re not just offering a seamless user experience but also signaling to search engines that your content is reliable and trustworthy. 

This proactive approach helps to ensure your site remains visible and valuable amidst ever-evolving ranking criteria.

With Ahrefs Site Audit tool, you can run a thorough SEO audit to uncover your website’s technical and on-page SEO issues, and find out exactly what’s holding your website back.

Plus, the platform recently added some exciting new features to enhance your analysis efforts.

Here are some key updates you should know about:

  • Core Web Vital (CWV) metrics: Filter your pages by various CWV data points during website recrawls. Visualize CrUX and Lighthouse scores, historical changes, and individual metric performance in the Performance report.
  • Google Data Studio integration: Build personalized reports by blending data from different sources and visualizing everything in the form of reports and dashboards.
  • Search by HTML code and page text: Easily search for specific elements extracted during a crawl, such as Google Analytics IDs or “out of stock” labels.
  • List of issues for a particular URL: Addressing an issue on a specific URL is streamlined with a dedicated tab showcasing all related issues, allowing comprehensive fixes.
  • Links Report in Site Audit Issues: Navigate issues more effectively with an additional links report tab, facilitating in-depth analysis at the links level. For instance, browse and export links related to 404 pages for a thorough understanding.

Easy Way: Use An SEO Plugin To Help Optimize Your Content

Ahrefs also launched a WordPress SEO plugin to further assist in your optimization efforts.

With this new plugin, you can automate content audits and see how each article performs for a chosen target keyword.

The tool also provides recommendations on how you can improve the article for better results.

Here are a few key capabilities:

Ready to get your site in tip-top shape for the year ahead? Check out Ahrefs’ latest offerings today.

How To Execute & Measure Your SEO KPIs More Efficiently

Keeping your website up-to-date with SEO trends is important – but the buck doesn’t stop there.


Staying competitive requires you to keep a vigilant eye on your rival sites as well, dissecting their strategies, and adapting your own accordingly.

Success in SEO isn’t just about visibility; it’s about outperforming your peers.

With Ahrefs, you can use the Site structure report in Site Explorer to quickly see how a competing website is structured, and which parts generate the most organic traffic.

Reassessing the competitive landscape and recalibrating your strategy based on real-time insights are the secrets to staying ahead of the curve.

Site structure is not the only noticeable update to Ahrefs Site Explorer tool. Its overview report has also been updated and is worth exploring.

Not only does it load noticeably faster than the previous version, but you get access to the following new features:

  • New history chart.
  • Comparison mode.
  • Paid traffic data.
  • Year-over-year mode.
  • Google algorithm updates.
  • Calendar report.
  • Site structure report.

Start Ranking Higher & Maximizing Opportunities With Ahrefs Now

In the rapidly evolving SEO landscape, Ahrefs emerges as a strategic ally, providing the tools you need to not only stay afloat but rise to new heights.

From leveraging AI tools intelligently to mastering technical SEO and executing with precision, we’ve dissected the key strategies that will redefine your SEO in 2024 and beyond.

It’s time to embrace change and elevate your search performance.

Start your journey to sustained SEO success with Ahrefs’ vast array of tools and exciting new features.

With their updated usage-based pricing model, you can access the features you need most. Try it today!

TRY IT NOW

Image Credits

Featured Image: Image by Ahrefs. Used with permission.

Ecommerce MGMT 0 Comments
Mar 18 2024
Website Health Checkup: Maximizing Technical SEO Performance In 2024 via @sejournal, @lorenbaker

In today’s digital landscape, a healthy, technically sound website is the key to SEO success. 

However, ensuring your site is optimized for maximum performance is no easy feat, as SEOs, developers, UX pros, content leaders, marketing, and C-suite members must all be in alignment.

Maintaining your website’s technical health isn’t just about ranking well in search engines – it’s also crucial to driving more user engagement, conversions, and ultimately, revenue.

On March 27, we’re delving into the critical areas and key site metrics you need to focus on in 2024, including SEO, site speed, and accessibility. 

Join us as we provide actionable tips and insights to help you improve your website’s technical health and drive tangible results.

And of course, we’ll go beyond just finding issues on your domains – we’ll also walk you through ways to actually fix them.

Our expert guests, Richard Barrett and Matt Ford of Lumar, will show you how to build new connected site health workflows that enable faster fixes and less stress for everyone involved in the website optimization process.

In this live session, we’ll cover the following topics:

  • Expanding Your View: Beyond Content and Keywords – Learn why SEO professionals and website managers need to broaden their perspective beyond content & keywords for maximum business results.
  • Improving Technical Health: Tips and Strategies – Discover practical tips for enhancing key aspects of your website’s technical health, including crawlability, indexability, site speed, accessibility, and more.
  • Building Streamlined Workflows: Aligning Your Team for Success – Explore strategies for building connected website health workflows that facilitate faster issue resolution and reduced stress for your team.
Register Now

Ready to take the next step toward elevating your website’s health and unlocking the full potential of your SEO strategy?

This webinar has the expert insights and actionable tips you need to enhance workflow efficiency and boost your site’s overall performance.

You’ll walk away knowing how to get the right platforms and processes in place to improve your site’s most critical health metrics.

Plus, Richard and Matt will be answering your questions live, following the presentation. 

Don’t miss this opportunity to start mastering your website’s technical health. Sign up now and secure your SEO success for 2024 and beyond.

Ecommerce MGMT 0 Comments
Mar 13 2024
How to optimize your crawl budget

Google doesn’t always spider every page on a site instantly. Sometimes, it can take weeks. This might get in the way of your SEO efforts. Your newly optimized landing page might not get indexed. At that point, it’s time to optimize your crawl budget. In this article, we’ll discuss what a ‘crawl budget’ is and what you can do to optimize it.

What is a crawl budget?

Crawl budget is the number of pages Google will crawl on your site on any given day. This number varies slightly daily, but overall, it’s relatively stable. Google might crawl six pages on your site each day; it might crawl 5,000 pages; it might even crawl 4,000,000 pages every single day. The number of pages Google crawls, your ‘budget,’ is generally determined by the size of your site, the ‘health’ of your site (how many errors Google encounters), and the number of links to your site. Some of these factors are things you can influence; we’ll get to that in a bit.

How does a crawler work?

A crawler like Googlebot gets a list of URLs to crawl on a site. It goes through that list systematically. It grabs your robots.txt file occasionally to ensure it’s still allowed to crawl each URL and then crawls the URLs individually. Once a spider has crawled a URL and parsed the contents, it adds new URLs found on that page that it has to crawl back on the to-do list.

Several events can make Google feel a URL has to be crawled. It might have found new links pointing at content, or someone has tweeted it, or it might have been updated in the XML sitemap, etc., etc… There’s no way to make a list of all the reasons why Google would crawl a URL, but when it determines it has to, it adds it to the to-do list.

Read more: Bot traffic: What it is and why you should care about it »

When is crawl budget an issue?

Crawl budget is not a problem if Google has to crawl many URLs on your site and has allotted a lot of crawls. But, say your site has 250,000 pages, and Google crawls 2,500 pages on this particular site each day. It will crawl some (like the homepage) more than others. It could take up to 200 days before Google notices particular changes to your pages if you don’t act. Crawl budget is an issue now. On the other hand, if it crawls 50,000 a day, there’s no issue at all.

Follow the steps below to determine whether your site has a crawl budget issue. This does assume your site has a relatively small number of URLs that Google crawls but doesn’t index (for instance, because you added meta noindex).

  1. Determine how many pages your site has; the number of URLs in your XML sitemaps might be a good start.
  2. Go into Google Search Console.
  3. Go to “Settings” -> “Crawl stats” and calculate the average pages crawled per day.
  4. Divide the number of pages by the “Average crawled per day” number.
  5. You should probably optimize your crawl budget if you end up with a number higher than ~10 (so you have 10x more pages than what Google crawls daily). You can read something else if you end up with a number lower than 3.
The ‘Crawl stats’ report Google Search Console

What URLs is Google crawling?

You really should know which URLs Google is crawling on your site. Your site’s server logs are the only ‘real’ way of knowing. For larger sites, you can use something like Logstash + Kibana. For smaller sites, the guys at Screaming Frog have released an SEO Log File Analyser tool.

Get your server logs and look at them

Depending on your type of hosting, you might not always be able to grab your log files. However, if you even think you need to work on crawl budget optimization because your site is big, you should get them. If your host doesn’t allow you to get them, it’s time to change hosts.

Fixing your site’s crawl budget is a lot like fixing a car. You can’t fix it by looking at the outside; you’ll have to open that engine. Looking at logs is going to be scary at first. You’ll quickly find that there is a lot of noise in logs. You’ll find many commonly occurring 404s that you think are nonsense. But you have to fix them. You must wade through the noise and ensure your site is not drowned in tons of old 404s.

Keep reading: Website maintenance: Check and fix 404 error pages »

Increase your crawl budget

Let’s look at the things that improve how many pages Google can crawl on your site.

Website maintenance: reduce errors

Step one in getting more pages crawled is making sure that the pages that are crawled return one of two possible return codes: 200 (for “OK”) or 301 (for “Go here instead”). All other return codes are not OK. To figure this out, look at your site’s server logs. Google Analytics and most other analytics packages will only track pages that served a 200. So you won’t find many errors on your site in there.

Once you’ve got your server logs, find and fix common errors. The most straightforward way is by grabbing all the URLs that didn’t return 200 or 301 and then ordering by how often they were accessed. Fixing an error might mean that you have to fix code. Or you might have to redirect a URL elsewhere. If you know what caused the error, you can also try to fix the source.

Another good source for finding errors is Google Search Console. Read our Search Console guide for more info on that. If you’ve got Yoast SEO Premium, you can easily redirect them away using the redirects manager.

Block parts of your site

If you have sections of your site that don’t need to be in Google, block them using robots.txt. Only do this if you know what you’re doing, of course. One of the common problems we see on larger eCommerce sites is when they have a gazillion ways to filter products. Every filter might add new URLs for Google. In cases like these, you want to ensure that you’re letting Google spider only one or two of those filters and not all of them.

Reduce redirect chains

When you 301 redirect a URL, something weird happens. Google will see that new URL and add that URL to the to-do list. It doesn’t always follow it immediately; it adds it to its to-do list and goes on. When you chain redirects, for instance, when you redirect non-www to www, then http to https, you have two redirects everywhere, so everything takes longer to crawl.

This is easy to say but hard to do. Getting more links is not just a matter of being awesome but also of making sure others know you’re awesome. It’s a matter of good PR and good engagement on social media. We’ve written extensively about link building; we’d suggest reading these three posts:

  1. Link building from a holistic SEO perspective
  2. Link building: what not to do?
  3. 6 steps to a successful link building strategy

When you have an acute indexing problem, you should first look at your crawl errors, block parts of your site, and fix redirect chains. Link building is a very slow method to increase your crawl budget. On the other hand, link building must be part of your process if you intend to build a large site.

TL;DR: crawl budget optimization is hard

Crawl budget optimization is not for the faint of heart. If you’re doing your site’s maintenance well, or your site is relatively small, it’s probably not needed. If your site is medium-sized and well-maintained, it’s fairly easy to do based on the above tricks.

Assess your technical SEO fitness

Optimizing your crawl budget is part of your technical SEO. Are you curious how your site’s overall technical SEO fits? We’ve created a technical SEO fitness quiz that helps you figure out what you need to work on!

Read on: Robots.txt: the ultimate guide »



Edwin Toonen

Edwin is a strategic content specialist. Before joining Yoast, he spent years honing his skill at The Netherlands’ leading web design magazine.

Avatar of Edwin Toonen

Coming up next!

Ecommerce MGMT 0 Comments
Mar 4 2024
8 Common Robots.txt Issues And How To Fix Them via @sejournal, @TaylorDanRW

Robots.txt is a useful and powerful tool to instruct search engine crawlers on how you want them to crawl your website. Managing this file is a key component of good technical SEO.

It is not all-powerful – in Google’s own words, “it is not a mechanism for keeping a web page out of Google” – but it can help prevent your site or server from being overloaded by crawler requests.

If you have this crawl block on your site, you must be certain it’s being used properly.

This is particularly important if you use dynamic URLs or other methods that generate a theoretically infinite number of pages.

In this guide, we will look at some of the most common issues with the robots.txt file, their impact on your website and your search presence, and how to fix these issues if you think they have occurred.

But first, let’s take a quick look at robots.txt and its alternatives.

What Is Robots.txt?

Robots.txt uses a plain text file format and is placed in the root directory of your website.

It must be in the topmost directory of your site. Search engines will simply ignore it if you place it in a subdirectory.

Despite its great power, robots.txt is often a relatively simple document and a basic robots.txt file can be created in seconds using an editor like Notepad. You can have fun with them and add additional messaging for users to find.

8 Common Robots.txt Issues And How To Fix ThemImage from author, February 2024

There are other ways to achieve some of the same goals that robots.txt is usually used for.

Individual pages can include a robots meta tag within the page code itself.

You can also use the X-Robots-Tag HTTP header to influence how (and whether) content is shown in search results.

What Can Robots.txt Do?

Robots.txt can achieve a variety of results across a range of different content types:

Webpages can be blocked from being crawled.

They may still appear in search results, but they will not have a text description. Non-HTML content on the page will not be crawled either.

Media files can be blocked from appearing in Google search results.

This includes images, video, and audio files.

If the file is public, it will still “exist” online and can be viewed and linked to, but this private content will not show in Google searches.

Resource files like unimportant external scripts can be blocked.

But this means if Google crawls a page that requires that resource to load, the Googlebot robot will “see” a version of the page as if that resource did not exist, which may affect indexing.

You cannot use robots.txt to completely block a webpage from appearing in Google’s search results.

To achieve that, you must use an alternative method, such as adding a noindex meta tag to the head of the page.

How Dangerous Are Robots.txt Mistakes?

A mistake in robots.txt can have unintended consequences, but it’s often not the end of the world.

The good news is that by fixing your robots.txt file, you can recover from any errors quickly and (usually) in full.

Google’s guidance to web developers says this on the subject of robots.txt mistakes:

“Web crawlers are generally very flexible and typically will not be swayed by minor mistakes in the robots.txt file. In general, the worst that can happen is that incorrect [or] unsupported directives will be ignored.

Bear in mind though that Google can’t read minds when interpreting a robots.txt file; we have to interpret the robots.txt file we fetched. That said, if you are aware of problems in your robots.txt file, they’re usually easy to fix.”

8 Common Robots.txt Mistakes

  1. Robots.txt Not In The Root Directory.
  2. Poor Use Of Wildcards.
  3. Noindex In Robots.txt.
  4. Blocked Scripts And Stylesheets.
  5. No Sitemap URL.
  6. Access To Development Sites.
  7. Using Absolute URLs.
  8. Deprecated & Unsupported Elements.

If your website behaves strangely in the search results, your robots.txt file is a good place to look for any mistakes, syntax errors, and overreaching rules.

Let’s take a look at each of the above mistakes in more detail and see how to ensure you have a valid robots.txt file.

1. Robots.txt Not In The Root Directory

Search robots can only discover the file if it’s in your root folder.

That’s why there should be only a forward slash between the .com (or equivalent domain) of your website, and the ‘robots.txt’ filename, in the URL of your robots.txt file.

If there’s a subfolder in there, your robots.txt file is probably not visible to the search robots, and your website is probably behaving as if there was no robots.txt file at all.

To fix this issue, move your robots.txt file to your root directory.

It’s worth noting that this will need you to have root access to your server.

Some content management systems will upload files to a “media” subdirectory (or something similar) by default, so you might need to circumvent this to get your robots.txt file in the right place.

2. Poor Use Of Wildcards

Robots.txt supports two wildcard characters:

  • Asterisk (*) – represents any instances of a valid character, like a Joker in a deck of cards.
  • Dollar sign ($) – denotes the end of a URL, allowing you to apply rules only to the final part of the URL, such as the filetype extension.

It’s sensible to adopt a minimalist approach to using wildcards, as they have the potential to apply restrictions to a much broader portion of your website.

It’s also relatively easy to end up blocking robot access from your entire site with a poorly placed asterisk.

Test your wildcard rules using a robots.txt testing tool to ensure they behave as expected. Be cautious with wildcard usage to prevent accidentally blocking or allowing too much.

3. Noindex In Robots.txt

This one is more common on websites that are over a few years old.

Google has stopped obeying noindex rules in robots.txt files as of September 1, 2019.

If your robots.txt file was created before that date or contains noindex instructions, you will likely see those pages indexed in Google’s search results.

The solution to this problem is to implement an alternative “noindex” method.

One option is the robots meta tag, which you can add to the head of any webpage you want to prevent Google from indexing.

4. Blocked Scripts And Stylesheets

It might seem logical to block crawler access to external JavaScripts and cascading stylesheets (CSS).

However, remember that Googlebot needs access to CSS and JS files to “see” your HTML and PHP pages correctly.

If your pages are behaving oddly in Google’s results, or it looks like Google is not seeing them correctly, check whether you are blocking crawler access to required external files.

A simple solution to this is to remove the line from your robots.txt file that is blocking access.

Or, if you have some files you do need to block, insert an exception that restores access to the necessary CSS and JavaScript.

5. No XML Sitemap URL

This is more about SEO than anything else.

You can include the URL of your XML sitemap in your robots.txt file.

Because this is the first place Googlebot looks when it crawls your website, this gives the crawler a headstart in knowing the structure and main pages of your site.

While this is not strictly an error – as omitting a sitemap should not negatively affect the actual core functionality and appearance of your website in the search results – it’s still worth adding your sitemap URL to robots.txt if you want to give your SEO efforts a boost.

6. Access To Development Sites

Blocking crawlers from your live website is a no-no, but so is allowing them to crawl and index your pages that are still under development.

It’s best practice to add a disallow instruction to the robots.txt file of a website under construction so the general public doesn’t see it until it’s finished.

Equally, it’s crucial to remove the disallow instruction when you launch a completed website.

Forgetting to remove this line from robots.txt is one of the most common mistakes among web developers; it can stop your entire website from being crawled and indexed correctly.

If your development site seems to be receiving real-world traffic, or your recently launched website is not performing at all well in search, look for a universal user agent disallow rule in your robots.txt file:

User-Agent: *

Disallow: /

If you see this when you shouldn’t (or don’t see it when you should), make the necessary changes to your robots.txt file and check that your website’s search appearance updates accordingly.

7. Using Absolute URLs

While using absolute URLs in things like canonicals and hreflang is best practice, for URLs in the robots.txt, the inverse is true.

Using relative paths in the robots.txt file is the recommended approach for indicating which parts of a site should not be accessed by crawlers.

This is detailed in Google’s robots.txt documentation, which states:

A directory or page, relative to the root domain, that may be crawled by the user agent just mentioned.

When you use an absolute URL, there’s no guarantee that crawlers will interpret it as intended and that the disallow/allow rule will be followed.

8. Deprecated & Unsupported Elements

While the guidelines for robots.txt files haven’t changed much over the years, two elements that are oftentimes included are:

  • Crawl-delay.
  • Noindex.

While Bing supports crawl-delay, Google doesn’t, but it is often specified by webmasters. You used to be able to set crawl settings in Google Search Console, but this was removed towards the end of 2023.

Google announced it would stop supporting the noindex directive in robots.txt files in July 2019. Before this date, webmasters were able to use the noindex directive in their robots.txt file.

This was not a widely supported or standardized practice, and the preferred method for noindex was to use on-page robots, or x-robots measures at a page level.

How To Recover From A Robots.txt Error

If a mistake in robots.txt has unwanted effects on your website’s search appearance, the first step is to correct robots.txt and verify that the new rules have the desired effect.

Some SEO crawling tools can help so you don’t have to wait for the search engines to crawl your site next.

When you are confident that robots.txt is behaving as desired, you can try to get your site re-crawled as soon as possible.

Platforms like Google Search Console and Bing Webmaster Tools can help.

Submit an updated sitemap and request a re-crawl of any pages that have been inappropriately delisted.

Unfortunately, you are at the whim of Googlebot – there’s no guarantee as to how long it might take for any missing pages to reappear in the Google search index.

All you can do is take the correct action to minimize that time as much as possible and keep checking until Googlebot implements the fixed robots.txt.

Final Thoughts

Where robots.txt errors are concerned, prevention is always better than the cure.

On a large revenue-generating website, a stray wildcard that removes your entire website from Google can have an immediate impact on earnings.

Edits to robots.txt should be made carefully by experienced developers, double-checked, and – where appropriate – subject to a second opinion.

If possible, test in a sandbox editor before pushing live on your real-world server to avoid inadvertently creating availability issues.

Remember, when the worst happens, it’s important not to panic.

Diagnose the problem, make the necessary repairs to robots.txt, and resubmit your sitemap for a new crawl.

Your place in the search rankings will hopefully be restored within a matter of days.

More resources:

Featured Image: M-SUR/Shutterstock

Ecommerce MGMT 0 Comments