Google Testing AI Tool That Finds & Rewrites Quality Content via @sejournal, @martinibuster

Google is paying small publishers to use their AI platform to rewrite the content published on news sites that are targeted by Google’s platform for largescale content parasitism. People on Twitter express unfavorable opinions about it.

Google is paying small independent publishers to use a generative AI platform that targets the content of other news content for summarization and republication. According to a paywalled article published on AdWeek, the tool is a beta test that requires the publishers to post a required number of articles per day in order to receive their payment.

Small publishers use a tool that shows them content chosen by the AI tool that can be selected in order to produce content at “at no cost.”

Why Google’s New Tool Is Problematic

It’s common for a news organization to pick up on news that’s broken by another news organization and if it’s done by a capable journalist they put their own spin on it. It’s the way things are done.

Google’s tool however appears to resemble a programmatic way to plagiarize content called article spinning. Article spinning is an automated tactic that uses website feeds to input published content from other sites that is then rewritten by the computer program, typically by replacing words with synonyms. AI however can spin content with greater nuance, essentially summarizing the content in a different tone by replacing entire sentences and paragraphs with content that’s the same as the original but expressed in a different manner.

What’s different is that this tool is something that Google itself is testing and that’s problematic not just because Google is the de facto gatekeeper for online content, the tool places a target on specific news organizations to have their content spun by small independent publishers.

On the one hand that could be a good thing because it could generate inbound links to the original publisher of the news. Free links, that’s a win-win, right?

But it’s not because news publishers don’t really benefit from links to content that has a shelf life of roughly 48 hours, at most. News is a hamster wheel of constant news publication in the service of keeping the wheel spinning to keep the business afloat. It’s a nonstop process that can easily be undermined by wholesale content dilution.

And that’s really at the hear of the problem with Google’s AI tool in that it dilutes the value that an organization creates by hiring professionals to create the “value add” content that Google frequently says it wants to publish. And that is what makes Google’s AI tool hypocritical at best and cynical at worst because Google is encouraging the creation of high quality content while simultaneously undermining it.

An army of publishers programmatically copying every published news article doesn’t look so good to the original publisher, especially if their content is overwhelmed by the parasitic AI in Google News, in the search results and by user preference for their local online news publisher who is republishing the news from the bigger publishers.

Reaction To Google’s AI News Tool

Technology journalist Brian Merchant (who writes for The Atlantic and has published a book) posted a virtual thumbs down on Twitter, a sentiment that was unanimously seconded.

He tweeted:

“The nightmare begins — Google is incentivizing the production of AI-generated slop.

If you are a news outlet who has accepted this meager deal, and especially if you are publishing AI-generated articles without disclaimers, you should be deeply ashamed.”

Brian followed up with this tweet with his observation of what’s wrong with Google’s AI tool for small news sites:

“If we in the media have learned ANYTHING from the last 10 years it is that we do not in fact have to settle for whatever scraps big tech throws us, and in fact it will screw us in the end—why would you participate in automating your field out of existence for like $30k a year???”

Merchant retweeted a comment by technology journalist Alex Kantrow:

“This is sad. Is this the web Google wants?”

Another person tweeted:

“It’s all about unchecked and untaxed profit.

Of course the is what Google wants – corporations selling out their employees and Americans in general for a quick buck.”

A person who works for Microsoft invoked the concept of “autophagy” which when an organism begins consuming itself such as when it is starving.

She tweeted:

“The quality of news content will decline and hurt search. Autophagy is a real threat to information quality and it looks like no one is taking it seriously.”

The Future Of Content

This isn’t just a “news” problem, it’s a problem for everyone who earns a living publishing online content. What can be used for News publications can easily be adapted for product reviews, recipes, entertainment and virtually any topic that affiliates publish content on.

How do you feel about Google’s new tool? Will it help small publishers compete against bigger sites or is it just the onset of autophagy in the body of online publishing?

Google Chrome Showing More Relevant Search Suggestions via @sejournal, @MattGSouthern

Google has released three new features for its Chrome web browser to provide more helpful and relevant search suggestions.

The features use Chrome’s access to aggregated user data to deliver crowdsourced recommendations.

Overview Of New Features

1. Search Suggestions Based On Previous Searches

Launching on the desktop version of Chrome, the first feature will provide tailored search suggestions in the browser’s address bar based on your previous searches and similar searches others have made.

For example, if you recently looked up the Korean dish “Japchae,” Chrome may suggest other popular Korean foods like Bulgogi, Bibimbap, and Tteok-bokki to help inspire your next search.

This crowdsourced approach allows Chrome to go beyond regurgitating a user’s search history and leverage the collective knowledge of what many others have searched for related to a given topic. The feature provides a way to discover new ideas and lateral topics you may have yet to consider.

2. Enhanced Shopping Search Suggestions

The second feature enhancement focuses on shopping-related searches on mobile.

Chrome will now display relevant product images alongside search suggestions for broader categories, whereas previously, images were only shown for searches specific to a particular product name.

This enables more visual browsing and inspiration for generic searches like “bohemian table” rather than needing to search for a specific table name upfront.

3. Improved Suggestions With Limited Connectivity

Lastly, Chrome’s improved on-device search capabilities will now provide search suggestions even with a poor internet connection on mobile.

By relying less on external server calls, users can get search ideas in Chrome when offline or with limited connectivity.

The feature also works in Incognito mode, providing suggestions while maintaining privacy.

Takeaways For Digital Marketers

For digital marketers and SEO professionals, these features highlight the increasing importance of ranking for valuable long-tail search queries.

As Chrome gets better at suggesting relevant ideas to users, optimizing content for specific searcher intent will become more critical.

Overall, these additions provide smarter search assistance to Chrome users while underscoring the need for brands to focus beyond just top keyword rankings.


FAQ

How will Google Chrome’s new search suggestion feature impact SEO strategies?

  • The introduction of tailored search suggestions based on aggregated user data and previous searches means SEO strategies must emphasize ranking for valuable long-tail queries that align with specific searcher intent.
  • With Chrome’s visualization of product images for broader shopping categories, SEO professionals should ensure products are optimized for textual and visual discovery.
  • Finally, as Chrome now provides search suggestions under limited connectivity conditions, brands should consider how their content can remain discoverable even in these scenarios, potentially targeting keywords likely to appear in these suggestions.

What are the practical SEO applications from Chrome’s enhancement of shopping search suggestions on mobile?

  • SEO professionals should focus on comprehensive image optimization, including relevant alt-text and high-quality images, to increase the likelihood of appearing alongside search suggestions for broader product categories.
  • Creating detailed product descriptions that utilize a mix of broad and specific keywords can also help products surface in these visually-driven suggestions.
  • For broader category terms like “bohemian table,” it’s advisable to include these phrases within titles, headers, and meta descriptions for enhanced visibility.

How can marketers prepare content for Google Chrome’s improved on-device search capabilities?

  • Marketers should optimize their content to be cache-friendly for scenarios with limited connectivity, using lightweight pages and efficient coding practices.
  • Ensuring high-quality, relevant content that can address a range of user queries will help be considered for Chrome’s on-device suggestions.
  • It’s also beneficial to maintain an up-to-date sitemap and use structured data to aid Chrome in understanding and retrieving content quickly, even in Incognito mode.

Websites Created With Google Business Profiles To Shut Down In March via @sejournal, @kristileilani

Do you have a website created through Google Business Profiles for your local business?

If so, you must find an alternative website solution as Google plans to shut down websites created with Google Business Profiles in March.

Websites Created With Google Business Profiles Will Redirect Until June 10, 2024

A redirect will be put in place from your GBP website to your Google Business Profile until June 10, 2024.

“Websites made with Google Business Profiles are basic websites powered by the information on your Business Profile.

In March 2024, websites made with Google Business Profiles will be turned off and customers visiting your site will be redirected to your Business Profile instead.

The redirect will work until June 10, 2024.”

How To Find Out If You Have A Google Business Profile Website

To find out if your business has a website made with Google Business Profile, search for my business or your business name on Google. Once you find your Google Business Profile, edit your profile and check for your website in the contact section.

If you have a Google Business Profile site, it should say, “You have a website created with Google.”

Otherwise, it will allow you to add the link to your website.

Websites Created With Google Business Profiles To Shut Down In MarchScreenshot from Google, February 2024

Choosing An Alternative Website Builders For Small Businesses

Google suggests Wix, Squarespace, GoDaddy, Google Sites, Shopify for ecommerce, Durable, Weebly, Strikingly, and WordPress as alternative website builders to create a new website or ad landing page to replace the Google Business Profiles site.

While some, like WordPress, offer a free website builder with generative AI features, its users’ content may reportedly be sold to OpenAI and Midjourney as training data unless they opt out.

Regarding Core Web Vitals, WordPress, Wix, and Squarespace showed the most improvements in performance.

It’s also worth noting that while Google Deepmind used a Google Sites website to introduce Genie, its new AI model, Google Sites may not be best for SEO.

Updating Ad Campaigns

If you have a Google Ads campaign that links to a website created with Google Business Profiles, the ad campaign will also stop running on March 1, 2024, until the website link is updated.

There’s still time to update your business website to ensure visitors are not sent to a 404 error page after June 10, 2024. If you haven’t chosen a new website builder or hosting service, review the reviews to find the most reliable, affordable, and optimized solution for your business.

Featured image: Vladimka production/Shutterstock

The Impact Of 5G Technology On Mobile SEO Strategies via @sejournal, @AdamHeitzman

The ongoing rollout of 5G wireless technology across the world brings mobile users much faster data speeds, highly reliable connectivity, and more widespread cellular coverage.

By 2028, the number of 5G subscriptions in North America is expected to reach 406 million.

But what does the rapid expansion of 5G mean for mobile SEO?

And what changes should you be making to your SEO strategy to stay up-to-speed with this latest generation of cellular network technology?

Let’s find out.

A Quick Tour Of 5G Technology

5G is the latest advancement in mobile internet and communication, significantly improving the speed, capacity, and efficiency of wireless networks.

The technology lets you download movies, games, and other large files in a matter of seconds, far outpacing its 4G predecessor with lightning-fast speeds of up to 1GB per second.

But 5G isn’t just about raw speed. It also allows several devices to connect to the network simultaneously without affecting performance.

This advancement is especially important as we move into an era where more and more devices will be connected to the internet, from smart home appliances to self-driving cars.

5G technology also drastically reduces network latency to as low as one millisecond (compared with 200 milliseconds for 4G).

In other words, there’s virtually no delay when you send or receive data.

Thanks to this reduction in lag, 5G allows for a more fluid and responsive user experience within real-time applications like online video games and video meetings.

In short, 5G is set to make the online world faster, smoother, and more connected than ever.

So, what will this mean for mobile SEO going forward?

5G And The Future Of Mobile SEO

Here are the five most important implications of 5G technology on SEO.

1. The Need For Speed

Page speed improves user experience and has been a Google ranking factor for mobile searches since 2018.

And since Google completed the switch to mobile-first indexing in October 2023, the mobile version of your site is essentially all that matters for rankings.

The advent of 5G doesn’t change the competitive advantages of optimizing your mobile site for speed.

Even though 5G means faster load times for websites across the board, the relative speed of individual websites still makes a difference. All else being equal, the fastest page will always win.

It’s also important to appreciate that people’s expectations around page speed will likely increase with 5G. Users will soon become accustomed to blazing-fast speeds and have little patience for slower-loading sites.

This means pages that fail to load quickly, even with 5G, will likely experience poor user engagement and, therefore, worse search engine visibility.

2. Location, Location, Location

5G networks provide better geolocation accuracy than 4G due to their higher frequency bands, denser network infrastructure, and advanced beamforming techniques.

As a result, 5G users can more easily access accurate information based on their location. This means local SEO will become an even greater priority for local businesses.

Optimizing for local search includes maintaining a complete and up-to-date Google Business Profile, targeting relevant location-based keywords, and ensuring your business contact details are consistent across the web.

3. More Emphasis On Rich Media

One of the standout features of 5G technology is its capacity to handle high-bandwidth content seamlessly.

This means websites will be able to incorporate more video content, high-resolution images, and interactive elements without encountering the performance issues that often arise on 4G networks.

Of course, using richer content formats opens the door to greater user engagement, thereby boosting your website’s ranking potential.

For example, if you’re an online retailer, you could incorporate 360 views of all your products or even interactive fitting rooms, allowing customers to try on their clothes using virtual avatars.

Likewise, if you run a science education website, you could use augmented reality to bring complex biological processes to life.

As businesses upgrade their SEO strategies to incorporate more immersive media formats, they’ll need to continue adhering to on-page optimization best practices to ensure search engines can correctly interpret and index the content.

For example, this means optimizing videos with accurate titles, relevant tags, and clear descriptions, as well as providing video transcripts.

4. Increased Use Of Voice Search

The rise of 5G is expected to significantly boost the use of voice search thanks to its faster speeds and reduced latency.

In fact, there were 142 million voice-search users in the U.S. in 2022. And this number is forecasted to reach 157 million by 2026.

As more people opt for the convenience of mobile voice search, the type of search queries they use will likely change. After all, voice search queries tend to be longer and more conversational than typed searches.

SEO strategies will have to adapt to this shift in user behavior by prioritizing long-tail keywords and creating more question-based content that mirrors how people use voice assistants.

Remember, many voice search responses are taken directly from Featured Snippets, which provide a direct answer to the user’s query within the search results.

Therefore, ensuring that pages contain comprehensive FAQ sections written in a natural, conversational tone will likely maximize your chances of being featured or chosen for voice search responses.

5. Technical SEO Considerations In A 5G World

Server Response Times

The advent of 5G technology sets a new benchmark for website performance.

Users accustomed to 5G’s rapid speeds will have little tolerance for slow-loading websites, making server response times a critical factor in user experience and SEO.

A sluggish server response can become a significant bottleneck, undermining the speed advantages offered by 5G.

Optimization Strategies

To capitalize on 5G’s potential, it’s essential to optimize server configurations.

Utilizing Content Delivery Networks (CDNs) can dramatically reduce latency by serving content from locations closer to the user.

Efficient backend processing, including streamlined database queries and optimized server-side scripting, can enhance response times.

Additionally, leveraging cloud-based solutions can provide scalability and reliability, ensuring consistent performance even during traffic spikes.

Image Optimization

In the 5G era, while the temptation to use high-resolution images is understandable, it’s crucial to strike a balance between image quality and website performance.

Overloading a site with large image files can still impact loading times and user experience.

Optimization Strategies

Employing responsive images that adapt to different screen sizes and resolutions is key.

Embracing next-gen image formats like WebP and AVIF, which offer superior compression and quality over traditional formats like JPEG and PNG, can significantly reduce file sizes without compromising visual fidelity.

Progressive Web Apps

PWAs are set to play a pivotal role in the 5G era, offering an app-like experience with the added benefits of faster loading times and enhanced functionality.

This aligns perfectly with the capabilities of 5G, leading to improved mobile user experiences and, indirectly, better SEO.

Optimization Strategies

Ensuring that PWAs are discoverable and indexable by search engines is crucial.

This includes making the content within the PWA crawlable and equipping the PWA with SEO-friendly features like meta tags, structured data, and accessible URLs.

Final Thoughts

5G technology looks set to revolutionize how businesses approach their mobile SEO strategies.

We’ve seen how faster upload and download speeds, greater connection capacity, and minimal latency will create new opportunities for more dynamic and interactive online experiences.

The best news is you can gain a head-start on the competition by proactively adapting your mobile SEO strategy to this inevitable shift.

More resources:


Featured Image: Lekhawattana/Shutterstock

Entities & Ontologies: The Future Of SEO? via @sejournal, @BennyJamminS

This is an excerpt from SEJ’s SEO Trends 2024 ebook, our annual roundup of expert opinions on what you can expect over the course of the next 12 months. 

If one takeaway will keep you on the cutting edge of SEO, it’s understanding Andrea Volpini’s quote about “ontologies.” But I haven’t seen anyone else present SEO this way, and there are key things to understand first, which are themselves complex:

  • Entities.
  • Schema.
  • Algorithm machine learning advancements.
  • What “helpful content” really means.
  • How large language models (LLMs) work.

I’ve organized several expert quotes to expand on the foundations you need to understand how websites must change to organize information effectively.

Then Andrea Volpini will finish the section by diving into “ontologies,” which will propel you into the future of SEO:

  • Standing out from AI outputs to build trust and nurture audiences.
  • Being discoverable within in-platform “search experiences” driven by AI.
  • Aligning your content organization and topical authority with how people and search engines understand concepts and entities.

If I had to summarize these insights in three sentences, they would be:

  • Understanding the “entity” is critical to future SEO success because it’s both a human and a data model of understanding.
  • Lean into exploration and knowledge by making meaningful connections.
  • Developing well-organized ontologies (maps of entities and their connections) will prepare you for success in AI platforms and Google Search, and improve user experiences.

Search Is Becoming A Human-Centered Exploration Tool

Angie Nikoleychuk, Behavioral Data Analyst, Search Engine Journal

Entities & Ontologies: The Future Of SEO?

Over the next year, I don’t think the big trend will be a tool or a ranking factor.

It will be a question: How do we meet the needs of AI while setting us apart from it?

The answer? Human-centered SEO for human-centered search – moving beyond ranking factors, RankBrain, and reviews to customer journeys and holistic strategies.

If the industry learned anything in the early days of article spinners and directories, it’s that optimizing for the technology of today is an expensive and temporary proposition. 

In 2024, we will see the SEO industry move toward optimizing where search engines and users are going. What’s driving this trend?

AI.

But not in the way you might think.

Standing Apart From AI: Human-Centered SEO

None of us are strangers to content written for search engines. In particular, AI-written content has increased the capacity for people to fill the web with low-effort, low-value content.

In response, I predict that search engines will release more and more spam and useful content updates to try and weed out low-value content.

Some argue that SGE-like search features will be the solution. Some SEO pros will attempt to optimize for SGE in its current form, but I expect changes to it before full integration with search.

Instead, I see the smart move being the trend toward human-centered search.

Of course, search engines are just one side of this digital coin.

Encountering more AI content that fails to meet the needs of end users will inevitably feed the steadily declining trust. As a result, users might shy away from more generic and AI-written content.

So, while this type of content might generate traffic, we will see this content genre start to generate fewer clicks and conversions – at least for YMYL and products and services that require more of a user’s time or money.

What are the best SEO strategies we’ll see appearing in 2024 in response to this? Human-centered ones.

They will consider where the user has been, where they’re going, and their motivations and interests – aspects that AI and search engine-focused content can’t touch. Here are a few examples:

  • Time and Type Segmentation: Where a particular target audience is. It’s a short window, but much like a roulette wheel, it is a gamble that will pay off. (If your keyword research includes the phrase “traffic funnels” or “AIDA,” you’re behind.)
  • Experiences: Not just meeting a need or solving a problem but sharing unique stories and views that turn pages from an online destination into an experience and journey. It’s thinking beyond the click and traffic.
  • Connections: The concept of “strings to things” is evolving into how and in what way those things and entities are connected. Partnerships, collaborations, and connections between entities that go beyond a simple link will be the things that matter.

Evidence for this trend can already be seen in Information Architecture, User Experience, Human-AI Interaction, Information Retrieval, and other related research area trends.

For example, patents and research around systems like the Knowledge Graph have been looking to classify and label relationships to capture the “how” and “why” things are connected for some time.

And those tiny details will turn search from a generic search tool into a human-centered exploration tool in the future.

The nuance of this shift cannot be overstated. The distinction between the SEO of the last few years and tomorrow’s human-centered search is small, but it’s important.

For years, keywords and links have defined what a page is about. In 2024, SEO and search will be about the “how.”


Entity Discovery & Followership

Jes Scholz, Marketing Consultant, JesScholz.com

Entities & Ontologies: The Future Of SEO?

One trend that often slips under the radar is the growing significance of building a followership on Google surfaces.

This functionality has long been part of Google News. Its influence on visibility is undeniable, offering visibility in the “Sources panel,” the “Following tab,” and the “From publishers you follow” SERP feature.

It is also now present in Google Shopping. By “favorating” specific brands, users set a preference filter on their shopping results, providing additional exposure.

Entities In Google Discover

The real game-changer is that this trend extends its reach into Google Discover. Users can click the bell icon in a Knowledge Panel to follow an entity as a topic of interest within Discover.

Content related to the followed entity then gains prime placement, often appearing at or near the top of Discover in a dedicated “following” feature box.

Discover is already a dominant force on mobile, boasting a dedicated following tab.

Moreover, it’s currently undergoing trials on the Google desktop homepage.

A single placement in Discover could outperform a month’s worth of traditional blue link traffic due to users’ inherent buy-in and interest.

This underscores the critical importance of entity optimization, which is how you can establish your brand in the Google Knowledge Graph and become eligible for entity followership.

To explore the future of followership, get a sneak peek by going into Google Chrome flags and enabling the upcoming follow features and changes to the New Tab Page (NTP).

I predict entity followership will emerge as a brand-level E-E-A-T signal in 2024.


Standardize & Organize Information Based On Meaningful Connections

Martha van Berkel, CEO, Schema App

Entities & Ontologies: The Future Of SEO?

Since 2011, Google has urged organizations to adopt Schema Markup (a.k.a. structured data) to help search engines better understand website content.

This also allows you to qualify for rich results to stand out in search.

Over the years, organizations have successfully leveraged schema markup to achieve rich results for products, reviews, FAQs, how-to, videos, and more, resulting in higher click-through rates (CTR) and more engaged, qualified traffic to their website.

However, the Google search landscape has evolved further in the past two years.

Schema Is More Than Rich Results: It’s Understanding

In 2022, we saw the performance of video, FAQ, and other rich results change throughout the year.

In 2023, Google changed the eligibility for video-rich results, deprecated how-to rich results from the SERP, and reduced the frequency that the well-loved FAQ-rich results were awarded.

It also released the new AI-powered Search Generative Experience (SGE) in experimental environments.

Other rich results still exist and continue to perform on Google.

But with AI-powered search engines, marketers need to rethink their strategy to stand out in search, thinking beyond rich results due to their volatility.

Instead, marketers should leverage the semantic nature of schema markup and implement it across their sites to help search engines understand how their content relates to searchers’ queries.

In October 2023, John Mueller, Search Advocate at Google, released a search update video sharing how rich results will continue to be added and removed.

He reiterated that schema markup is a “machine-readable code that search engines use to understand content.” He suggested using a schema markup solution to update your structured data easily.

Good news! Schema markup – the same SEO strategy that helps machines understand and achieve rich results – can help you be understood and perform well in the new world of AI search.

Why? Because search engines are shifting from lexical (matching keywords) to semantic search (matching meaning). Schema markup helps define content explicitly, enabling search engines to understand your content’s meaning.

How does this work? By implementing schema markup on your site, you translate your content using the standardized Schema.org vocabulary and define the relationship between things on your site.

This provides specified context about your concepts and how they relate to other things on your site and the web.

As a result, machines can better understand your content, make factual inferences about your organization, and provide searchers with accurate results.

How Structured Data Can Improve CTR

At Schema App, we’ve recently seen a general decline in impressions and clicks across our customers due to rich result changes from Google.

However, we’ve also seen an increase in CTR after applying robust, connected schema markup.

Bing’s Principal Program Manager, Fabrice Canel, spoke at this year’s Pubcon. I infer from his presentation that he also expects new search experiences to improve user experiences as they are introduced.

If markup is managed correctly, users get contextual information in the search engine results page (SERP) before clicking through to your site.

That said, the world of AI search is still riddled with unknowns. We don’t know how it will change top results on the SERP, we don’t have any specific metrics from Google (we do from Bing), and we don’t know how quickly things will change.

So, focus on what you can control, like developing unique content and adding schema markup to be understood by machines.

Let’s capture those clicks, take control, and get more leads by speaking the language of search engines.


Entities Are The New Links

Ulrika Viberg, CEO and Senior SEO, Unikorn

Entities & Ontologies: The Future Of SEO?

We will see growth in AI, but not only in search!

We will see more AI as enriched data for products, such as more creative product categories based on specific situations.

A category for books to read in the winter can list horror stories, romance, short novels, and classics.

These lists are traditionally curated manually; with AI, they can be generated on the fly efficiently, and SEO pros will love it.

Things will get technical again; On-page will expand to not only H1s and alt-tags, but also, to nested structured data and entity linking, building structured data, and entity link graphs.

With AI, my gut says links will be less important.

Entities are the new links. The Google team has already mentioned that links are no longer as important as they used to be, and I believe AI will solve the “trusted websites” situation without links.


Beyond Entities: Building Maps Of Meaning With Ontologies

Ben Steele, Managing Editor, SEJ

We’ve built the foundation, so it’s time to dig into ontologies. Andrea Volpini provides exceptional insight, but let’s get through a few definitions first.

Ontologies aren’t just a trend; they’re an advanced way of understanding:

  • Your website site and content.
  • How people make meaningful connections.
  • How algorithms make meaningful connections.

Before we discuss how ontologies relate to search, it helps to understand the various meanings of the word ontology.

Ontology is a discipline of both philosophy and information science.

  • As a branch of metaphysics, ontology is most concerned with being, existence, and how beings and events relate to one another on their most basic level.
  • In information science and data management, ontologies are “explicit formal specifications of the terms in the domain and relations among them.”

In plain language and for our purposes, “ontology” studies entities. It categorizes types of entities and seeks to understand how they relate to each other.

The critical point I want you to see here is that the data management version of ontology owes its existence to the philosophy of ontology. Both are about how humans experience and categorize entities.

An ontology for SEO is detailed and explicit, both of each individual entity and the categorization system it falls under.

Of course, defining a narrow category isn’t always easy. The lines get blurry, and different systems overlap, merge, or split into new categories.

Ontologies aren’t pillars; they’re critical to information architecture and semantic technology.

Creating deep maps of connection and meaning is beyond the current capability of large language models (LLMs). But they do use them.

Modern AI-driven algorithms use human philosophical concepts to make connections.

More than standing out from LLMs, constructing ontologies will also help the models return accurate information about your website, as Volpini explains. Understand them now to prepare for Search Generative Experience (SGE).

These are my best tips for functionally applying what you learn from Volpini’s insightful essay below:

  • It can help to think of an ontology as a map of meaning: Create content with an understanding of its place within an ontology and all of the important entities within the content itself and the wider meaning map; then, build those connections using the SEO tools available to you, such as linking, schema, category organization, and site hierarchy.
  • Fully understanding entities and applying ontologies to all levels of your website, from your top menu navigation to every piece of content you create, is the path forward. Building websites for ontologies allows you to: Ensure your whole website is helpful and easy to navigate for users; speak the language of search engines and encourage more robust and accurate indexing of your pages; help LLMs in “search experience” features return accurate information about your website, surfacing your brand in generative results.

Ontologies In SEO

Andrea Volpini, Co-Founder And CEO, WordLift

Entities & Ontologies: The Future Of SEO?

In an era where artificial intelligence becomes increasingly intrinsic to SEO strategies, entities and Google’s Knowledge (and Shopping) Graph have emerged as pivotal elements.

They bridge the nexus between Search Generative Experience (SGE) and Google’s enhanced ecommerce initiative.

With a surge in AI-generated content and the dawn of AI authorship, you need to develop intricate ontologies if you want to stand out.

Schema markup gives search engines a better understanding of the content on a webpage. Ontologies take this a step further.

With ontologies, marketers and SEO professionals can define more complex relationships and hierarchies within their domain of knowledge.

This leads to better content organization, more relevant search results, and improved quality when the brands interact with large language models and the generative AI stack.

These ontologies transcend the mere facilitation of seamless interactions with LLMs; they will expand schema markup and, crucially, preserve and amplify a brand’s unique voice amidst the vast sea of AI-generated content.

Ontologies Will Act As The Discernible Differentiator In The Digital Realm

Prompting techniques are evolving. Entities – both named entities and concepts – are having a profound influence on transferring knowledge to language models.

This paves the way for the automation of prompt creation and the models’ autonomous learning of optimal prompting strategies.

Furthermore, integrating autonomous AI agents will be strategic for exhaustive SEO automation, where multiple SEO tasks can be conducted with limited human intervention – from dynamic internal links to crawling optimization strategies, from on-page SEO to schema markup.

As these trends will shape SEO, the centrality of entities and ontologies is not merely advantageous but indispensable.

The Rise Of Comprehensive And Interactive Search Experiences

Navigating through the Generative AI epoch, Search has witnessed a transformative evolution, particularly in rendering a more comprehensive and interactive user experience.

Bing notably paved the way in 2017, introducing chatbot integration into Search and enhancing user interactions through innovative conversational interfaces.

Bing Chat and Google SGE are not swift adaptations but a culmination of decades of technological evolution and strategic adaptations in voice search and interactive result exploration.

The transition from “Hey Google” to “Hey Bard” could make Search directly accessible to millions of users, creating new opportunities to build personalized digital experiences.

Anyone with a Google Assistant device will be able to access generative AI and search through voice.

A noticeable shift in search traffic and the increase of zero-click searches, which reached about half of Google searches in 2022, spotlights a transition towards providing immediate answers on search engine result pages.

We’re heading for seamless, interactive, and enriched search experiences, shaping the future trajectory of online user engagement and content discoverability.

I do expect new opportunities with voice search, but overall, a way more competitive environment and a potential drop of up to 15-20% of search traffic on informational queries.

SEO could become more about activating user experiences tied to your brand inside search platforms than driving clicks.

This will require a significant adjustment in how you think about and track SEO performance.

I will leave you with the following that I wrote for an internal presentation:

“SEO is becoming a new frontier in an AI-driven society. It is the bridge between human knowledge and machines.

As we develop more and more AI-powered technologies, it is important to remember that we all share the responsibility of creating an AI that follows well-defined ethical guidelines and respects fundamental human values.

This includes individual rights, privacy protection, non-discrimination, fairness, and non-manipulation.”

More resources: 


Featured Image: Paulo Bobita/Search Engine Journal

SEO For Paws Live Stream Conference To Support Pet Shelters In Ukraine via @sejournal, @theshelleywalsh

One of the industry’s best-known influencers is organizing SEO for Paws, a live-streamed fundraiser on February 29, 2024, featuring a stellar speaker list that includes some of the industry’s best SEO professionals.

Anton Shulke is best known as the head of influencer marketing at Duda and an expert at organizing live-stream events.

When the war broke out in Ukraine, Anton was living in Kyiv but managed to seamlessly continue his work running live events while also escaping the conflict in Kyiv with his family and cat.

Even though Anton managed to leave the city, he has tirelessly continued his support for his favorite charity, which aids the many pets that were left behind in Kyiv after the war broke out.

SEO for Paws will mark two years since war broke out in Kyiv, and Anton has organized the event to provide even more support to the charity.

All proceeds will go to animal shelters in Ukraine that care for cats and dogs.

An Inspirational Journey With His Family And Dynia The Cat

When war broke out, Shulke was living in Kyiv with his wife, children, and cat, Dynia.

As the situation rapidly evolved, Anton kept his friends informed of his situation with his daily #coffeeshot posts on social media.

Anton Shulke And Dynia The CatImage from Anton Shulke, February 2024

As the situation in Kyiv became more serious, his frequent updates became a lifeline for his SEO friends and network, who anxiously awaited news of his safety on a daily basis.

After managing to get out of Ukraine and traveling across Europe, it was actually Dynia who chose the final destination of Spain, where Anton sought asylum with his family.

Though facing personal hardships, Shulke has remained dedicated to Ukraine’s small, donation-dependent pet shelters and vulnerable animals.

Shulke shares:

“Before the war, I tried to help those small cats and dogs shelters, but just a bit.

We are talking about super small shelters, 30-100 animals, they are in private flats. Sometimes in tiny flats, like one-bedroom or even studio flats. And the owner lives there; often, it is family.”

These tiny shelters operate entirely on donations as they do not receive government funding or support from large charities.

The war has made their situation even more dire, with increased animal abandonment and limited resources.

SEO For Paws – Cat Lovers, Dog Lovers, And SEO

The upcoming “SEO for Paws” livestream aims to continue fundraising efforts. The five-hour event, which runs from 10:55 a.m. to 3:30 p.m. ET, will offer actionable SEO and digital marketing advice from experts while raising money for the animal shelters.

Headline speakers who have donated their time to support his cause include Navah Hopkins, Dixon Jones, Ashley Segura, Barry Schwartz, Glenn Gabe, Arnout Hellemans, and Grant Simmons, among others.

Attendance is free, but participants are encouraged to donate.

Event Highlights

  • Date and Time: February 29, 2024, from 10:55 a.m. to 3:30 p.m. ET (4:55 p.m. to 9:30 p.m. GMT).
  • Access: Free registration with the option to join live, participate in Q&A sessions, and network with peers. A recording will be made available on YouTube.
  • Speakers: The live stream will feature 17 SEO and digital marketing experts, as well as some “special furry guests,” who will share actionable insights into SEO strategies, PPC tips, and content creation hacks.
  • Networking Opportunities: Attendees will have the chance to interact with experts and colleagues through a live chat during the event.

The event page states:

“Join us for an extraordinary live stream dedicated to supporting Ukrainian cat and dog shelters while diving deep into the world of SEO and digital marketing. Whether you’re a cat person, a dog lover, or simply passionate about SEO, there’s something here for everyone!”

How To Make A Difference

The “SEO for Paws” live stream is an opportunity to make a meaningful difference while listening to excellent speakers.

All money raised is donated to help cats and dogs in Ukraine.

You can register for the event here.

And you can help support the charity by buying coffee.

Search Engine Journal is proud to be sponsoring the event.

More resources: 


Featured Image: savitskaya iryna/Shutterstock

Universe: A Better Alternative To Keyword Research via @sejournal, @Kevin_Indig

Boost your skills with Growth Memo’s weekly expert insights. Subscribe for free!

The classic approach to keyword research is outdated. Instead of doing a sprint every so often, build a keyword universe to explore and conquer.

To Infinity And Beyond

A Keyword Universe is a big pool that surfaces the most important keywords at the top and lives in a spreadsheet or database like BigQuery.

A Keyword Universe is a big pool that surfaces the most important keywords.Image Credit: Kevin Indig

The goal is a keyword pipeline for content creation prioritized by business impact.

Keyword Universes elevate the most impactful topics to the top of a list, which allows companies to focus on planning capacity, like the number of published articles.

A Keyword Universe is a big pool that surfaces the most important keywords.Image Credit: Kevin Indig

A big problem in SEO is knowing which keywords convert to customers before targeting them. The Keyword Universe mitigates that problem through a clever sorting system.

One big advantage of the Keyword Universe compared to research sprints is that new keywords automatically fall into a natural prioritization.

SEOs can continuously research and launch new keywords into the universe, while writers can pick keywords off the list at any time. Fluid collaboration.

A Keyword Universe is a big pool that surfaces the most important keywords.Image Credit: Kevin Indig

Keyword Universes are mostly relevant for companies that have to create content themselves instead of leaning on users or products. I call them Integrators. Typical culprits are SaaS, DTC, or publishing businesses, which have no pre-determined structure for keyword prioritization.

The opposite is Aggregators, which scale organic traffic through UGC or product inventory.

Examples are TripAdvisor, Uber Eats, TikTok, and Yelp. The keyword path for Aggregators is defined by their page types. The target topics come out of the product.

Yelp, for example, knows that “near me keywords” and query patterns like “{business} in {city}” are important because that’s the main use case for their local listing pages.

Integrators don’t have that luxury. They need to use other signals to prioritize keywords for business impact.

Building a Keyword Universe is a three-step process:

Step 1: Mine

Keyword mining is the science of building a large list of keywords and a bread-and-butter workflow in SEO. The classic way is to use a list of seed keywords and throw them into third-party rank trackers to get related terms and other suggestions. That’s a good start.

But don’t ignore other valuable sources of keyword ideas:

  • Customer conversations: terminology and topics brought up by customers.
  • SERP features like PAAs and Google Suggest.
  • Search Console: keywords Google tries to rank but doesn’t have the right content have high impressions but low clicks.
  • Competitor ranks and paid search keywords.
People Also Search For feature in Search ResponsePeople Also Search For feature in Search Response (Image Credit: Kevin Indig)
Semrush’s list of paid keywords a company bids onSemrush’s list of paid keywords a company bids on (Image Credit: Kevin Indig)

The goal of the first step is to bloat our Universe with as many keywords as we can find. Don’t obsess over relevance. That’s Step 2.

Step 2: Align

Step 2, sorting the long list of keywords, is the linchpin of Keyword Universes. Getting it right is installing a powerful prioritization system for your company. Getting it wrong is wasting time. Anyone can create a large list of keywords, but creating strong filters and sorting mechanisms is hard.

The old-school way to go about prioritization is by search volume. We can do better than that. Most times, keywords with higher search volume actually convert less well.

Since we don’t know which keywords are most important for the business before targeting them, we need sorting parameters based on strong signals.

We can summarize several signals for each keyword and sort the list by a total score. That’s exactly what I’ve done with clients like Ramp, the fastest-growing fintech startup in history, to prioritize content strategy.

Real example of a Keyword UniverseReal example of a Keyword Universe (Image Credit: Kevin Indig)

Sorting is about defining an initial set of signals and then refining it with feedback. We start by giving each signal a weight based on our best guess and refine it over time.

Potential signals (not all need to be used):

  • MSV (monthly search volume).
  • KD (keyword difficulty) / competitiveness.
  • (MSV * KD) / CPC → I like to use this simple formula to balance search demand with competitiveness and potential conversion value.
  • Traffic potential.
  • Conversions from paid search or other channels.
  • Keyword is mentioned in customer conversation.
  • Keyword is part of a topic that converts well.
  • Growing or shrinking MSV.
  • Query modifier indicates users are ready to take action like “buy” or “download.”

We give each signal a weight from zero to 10 or zero to three, with the highest number being strongest and zero being weakest. The sum of all scores determines the sorting of the list. Keywords with the highest total score land at the top and vice versa. New keywords on the list fall into a natural prioritization.

Step 3: Refine

Models get good by improving over time. Like an LLM that learns from fine-tuning, we need to adjust our signal weighting based on the results we see. We can go about fine-tuning in two ways.

Anecdotally, conversions should increase as we build new content (or update existing content) based on the Keyword Universe. Otherwise, sorting signals have the wrong weighting, and we need to adjust.

Another way to test the system is a snapshot analysis: a comparison of the keywords that attract most organic clicks to the pages that drive the most conversions with the keywords at the top of the Universe. Ideally, they overlap. If they don’t, aim to adjust your sorting signals until they come close.

Tips

1. It’s easy to lose overview when mixing keywords that led to new content with keywords that haven’t yet. Move keywords that were written about to a different list and monitor ranks in a third-party rank tracker when operationalizing the Keyword Universe. I suggest a second tab on a spreadsheet or another table in a database.

2. Build custom click curves for each page type (blog article, landing page, calculator, etc.) when including traffic and revenue projections. Assign each step in the conversion funnel a conversion rate, like visit → sign-up, and multiply search volume with an estimated position on the custom click curve, conversion rates, and lifetime value. Fine-tune regularly.

Example: MSV * CTR (pos 1) * CVRs * Lifetime value = revenue prediction

3. GPT for Sheets or the Meaning Cloud extension for Google Sheets can speed up assigning each keyword to a topic. Meaning Cloud allows us to easily train an LLM by uploading a spreadsheet with a few tagged keywords. GPT for Sheets connects Google Sheets with the OpenAI API so we can give prompts like “Which of the following topics would this keyword best fit? Category 1, Category 2, Category 3, etc.”

Worldviews

Having the right filter is not just a concept for SEO but for life. Our sorting signals are worldviews, principles and values shaped by experience.

Prioritizing what’s important (sorting) allows us to literally filter through the noise (distractions, offers, shiny objects) and bring us to where we want to be. But they can’t improve without fine-tuning with feedback.

For a long time, I operated with the filter that managing a lot of people was the optimal way to respect and money.

Turns out, it’s really not. But it took me years and a few painful lessons to come to that realization.

Those lessons and reflecting on them are the feedback mechanisms that help me fine-tune my filter and define better priorities, just like adjusting the weight of sorting signals in our Keyword Universe surfaces better target keywords.

The key to using signals is being aware of which signals impact our decisions and using feedback to fine-tune them.


Featured Image: Lyna ™

Google Confirms That Filetype Search Operator’s Missing via @sejournal, @martinibuster

Google’s filetype search operator went missing early Wednesday morning. Other search operators continue to function. SearchLiaison responds that it must be a bug.

Filetype Search Operator

The filetype: search operator is a way to search for results that are constrained to specific kinds of files like text, pdf, .doc files and many other kinds of files.

@jeroenbosman on Twitter called attention to the fact that the search operator stopped working several days ago on February 26th.

They tweeted:

“Two Google (search) options have apparently been switched off: searching by filetype and seeing the cache. The former will be missed especially, and searching for e.g. (policy) documents will become much harder because of it.”

But it must have subsequently been fixed because it was working on Tuesday evening because I used it. But it must have gone down in the United States a few hours later.

But then early morning Wednesday (Eastern Time) SearchLiaison tweeted a response that it must be a bug.

He tweeted:

“This is probably a bug on filetype, and we’re looking into it.”

Screenshot Of Missing Filetype Search Results

Google Confirms That Filetype Search Operator’s Missing

How To Make Filetype Operator Work

If the filetype search operator hasn’t been fixed later Wednesday, there is a way to bring it back.

The person who first reported the outage tweeted:

“One correction (HT @lawliberg) : you can still use filetype: as long as you also restrict to a domain using the site: command. Still a strange and deplorable restriction, just as the strange restriction that you cannot use verbatim in combination with date filter.”

That tweet was correct, combining the filetype: search operator with another search operator like the site: search operator makes the filetype: search operator function again.

Below is a screenshot of a search for a Microsoft Word document in the .doc file type, constrained to the .gov top level domain (TLD).

Google Confirms That Filetype Search Operator’s Missing

The Filetype Search Bug Is Spreading?

There were reports by people from around the world who were tweeting about the filetype: search issue several hours earlier. It remains to be seen how serious this bug is and how long it will last.

Featured Image by Shutterstock/photosince

SEO Isn’t Just SEO via @sejournal, @coreydmorris

Investing in SEO resources internally with consultants, software, or agencies with specific organic search goals in mind, is awesome.

What isn’t awesome is budgeting for and  “buying” something and then finding out that you need to pay more and buy even more things later.

Everyone hates change orders, new budget requests, and having to go back and ask for more than they thought was needed.

If you’re an in-house SEO or marketing manager – or someone who has accountability over digital marketing performance that includes SEO – you need to know the full picture of what is necessary to be successful and ensure that your marketing plan includes all the resources needed to meet your goals.

SEO isn’t something that can be done in a silo, on an island (metaphorically), or alone. It requires a lot of other disciplines and decision-makers.

In this article, I will share the 10 things you need to consider so you don’t get surprised with change orders or new expenses down the road in order to reach your SEO goals.

1. Executive & Management Support

As I unpack all the things that go into SEO that don’t typically include a job title that has “SEO” in it, it is important to start with the fact that your management or leadership team needs to fully understand and support SEO.

You must make sure they understand that SEO isn’t just SEO; they should have a grasp of the complexities of SEO and other necessary expenses, at least to the degree required for you to be successful.

That includes understanding reasonable goals, expectations, and resource requirements to be effective.

You don’t have to have a boss who is an SEO nerd, but you do need them to be aware and supportive of your full-scale needs, which includes the items that follow on this list – and maybe more.

2. Developed Products Or Services

With the investment of time, technology, and expertise that SEO requires, it is hard to do if there are gaps in product development, a lack of demonstrated sales, or a brand that lacks strategy and clarity.

Without any of those key ingredients, you might have a moving target or find yourself trying to achieve unreasonable goals.

The website you are optimizing and driving traffic to and the content you create for it can take a lot of work.

You don’t want to waste that work focusing on the wrong topics and content relevance you’re trying to build.

It can be expensive to retest audiences or constantly change your targeting and goals, essentially having to start over all the time.

3. Legal & Compliance

I understand the reasons for legal reviews, guidelines, and compliance within industries and companies. Ensuring that everything is done right is important.

However, it is something that SEO pros don’t think about at times.

Whether it is knowing what you can and can’t publish on the site, the lead times needed, or the full process for getting content and web updates through the process, this is another area that isn’t exactly SEO, but can have a big impact on it.

Industries like healthcare, financial services, and much more can expect to have a pretty stringent legal and compliance process, and it has to be factored into SEO plans, including anything ranging from timing to costs involved.

4. Content

I know some super-talented SEO professionals who are great copywriters.

In smaller companies, people often wear multiple hats. So the same person might be working on SEO strategy and tactical implementation, including writing copy and content connected to the SEO strategy.

However, the art and science of creating content can often be well outside of the skillset of doing SEO.

When it comes to implementing the plan and seeing content all the way through to success for the full customer journey and goals/objectives, people with titles that don’t include SEO are often needed.

Understanding where content fits and how to invest in it outside of the SEO roles is crucial – whether it involves an in-house writer or an external resource –  as content is a big part of SEO.

5. UX

An SEO pro’s job typically isn’t finished when a user clicks through from a search engine result to the website.

In fact, that’s often when the excitement starts. The user is now on your site, moving through your funnel or customer journey towards the actions that will help them convert and reach your end goals.

Yes, some SEO pros, agencies, etc, only report on “SEO” metrics and stop there. That’s frustrating to me, as I see too many clients and companies left to connect the dots on their own.

Then, there’s a disconnect between SEO, UX, and web development.

If there’s no one thinking about or implementing any conversion rate optimization (CRO) testing and thinking on the website, then there’s a big gap between all of the teams.

Yes, we want the site to be beautiful, accessible, and engaging.

We have to merge together the content, page experience, and conversion flow thinking – and SEO folks aren’t typically given the keys to the site design, page layouts, and aesthetics (nor should they be  – I admit as an SEO professional).

6. Dev & IT

Often going hand in hand with UX is the need for people in dev and/or IT roles.

Some technical SEO pros know their way around the back end of sites and server configurations.

There’s nothing better than a deep, nerdy conversation about this subject matter.

However, even if the SEO expert knows their stuff in this area, most companies have things locked down, with dedicated in-house or contracted roles with checks and balances in place.

Even if you have a great website and all the content management system (CMS) controls you might want for doing on-page optimization, there seems to inevitably be a development need.

Whether it is the implementation of content architecture changes with moving pages, changing URL structures, and implementing redirects, or the need to optimize page experience, it can be a big blocker if you don’t have the budget or availability of these resources in your plan.

7. Sales

Sales teams and people don’t need to be drivers of SEO strategy and tactics.

However, the age-old disconnect between sales and marketing teams is still very real in many companies.

I encourage making an effort to include salespeople and leverage them in your work.

Getting on the same page is a baseline goal, but getting valuable feedback from salespeople on content that resonates with prospects, what SEO conversions are turning into “won” business, etc., is important to help shape your strategy and overall optimization plan.

8. Your Clients/Customers

Similar to how sales can be a great resource for downstream data to help shape your SEO efforts, you should also be talking to your clients or customers.

Even if you have the best data in your customer relationship management (CRM) or systems in place tracking a contact from first touch point to lifetime value, there’s still value in getting their perspective.

In a world where our cookies and data are going away, closing the loop is even more important.

Having a plan, communications, and incentives in place to get feedback from customers is vital so you can use that to refine your targeting, strategy, and better leverage content and teams involved with the website.

9. Tech Stack

SEO relies on technology to do a lot of heavy lifting.

I laugh when I think back to my start in SEO nearly 20 years ago and how many things I did manually back then in documents and spreadsheets that tools and technology do in minutes today.

The research tools you need, reporting platforms, and even the new and exciting AI tools all have costs attached to them.

I know I’m not talking about people here who are outside of SEO, but I’d be remiss if I didn’t mention your tech stack and needs, as you don’t want to be hamstrung by not being able to invest in the right tools you need to do the best SEO possible.

Make sure that those who are making the SEO investment decisions know that, beyond the other resources you need from other functions, you also need additional dollars for the tech stack.

10. Your Own Skills/Team Investment

Similar to the investment in your tech stack, if you’re an SEO pro, you’re going to want and need more resources to invest in yourself and your team.

SEO is moving as fast as ever (if not faster).

If you’re the only person/people in your organization who understand it, you’re in a vacuum.

You’re going to want to get or leverage personal development dollars to join peer networks, attend conferences, get expert training, or level up however you want and need to.

Conclusion

SEO requires a lot of people who don’t have titles that include “SEO.”

As an in-house person accountable for SEO, you need a lot of other functions that include people (and likely dollars as well) to get the job done and meet your goals.

Consider these eight specific functions and the last two areas of investment to make sure you’re prepared for the true cost of SEO. Whether you get to be part of a budgeting process, own the budget, or have expectations of how SEO will perform, it’s critical to have a full understanding from the start of a project.

Understand all the parts and pieces, the full cost of the effort, know your ROI equation, and run a smooth plan to get to your goals!

More resources:


Featured Image: baranq/Shutterstock

Google Releases New ‘How Search Works’ Episode On Crawling via @sejournal, @MattGSouthern

Google has published a fresh installment of its educational video series “How Search Works,” explaining how its search engine discovers and accesses web pages through crawling.

Google Engineer Details Crawling Process

In the seven-minute episode hosted by Google Analyst Gary Illyes, the company provides an in-depth look at the technical aspects of how Googlebot—the software Google uses to crawl the web—functions.

Illyes outlines the steps Googlebot takes to find new and updated content across the internet’s trillions of webpages and make them searchable on Google.

Illyes explains:

“Most new URLs Google discovers are from other known pages that Google previously crawled.

You can think about a news site with different category pages that then link out to individual news articles.

Google can discover most published articles by revisiting the Category page every now and then and extracting the URLs that lead to the articles.”

How Googlebot Crawls the Web

Googlebot starts by following links from known webpages to uncover new URLs, a process called URL discovery.

It avoids overloading sites by crawling each one at a unique, customized speed based on server response times and content quality.

Googlebot renders pages using a current version of the Chrome browser to execute any JavaScript and correctly display dynamic content loaded by scripts. It also only crawls publicly available pages, not those behind logins.

Improving Discovery & Crawlability

Illyes highlighted the usefulness of sitemaps—XML files that list a site’s URLs—to help Google find and crawl new content.

He advised developers to have their content management systems automatically generate sitemaps.

Optimizing technical SEO factors like site architecture, speed, and crawl directives can also improve crawlability.

Here are some additional tactics for making your site more crawlable:

  • Avoid crawl budget exhaustion – Websites that update frequently can overwhelm Googlebot’s crawl budget, preventing new content from being discovered. Careful CMS configuration and rel= “next” / rel= “prev” tags can help.
  • Implement good internal linking – Linking to new content from category and hub pages enables Googlebot to discover new URLs. An effective internal linking structure aids crawlability.
  • Make sure pages load quickly – Sites that respond slowly to Googlebot fetches may have their crawl rate throttled. Optimizing pages for performance can allow faster crawling.
  • Eliminate soft 404 errors – Fixing soft 404s caused by CMS misconfigurations ensures URLs lead to valid pages, improving crawl success.
  • Consider robots.txt tweaks – A tight robots.txt can block helpful pages. An SEO audit may uncover restrictions that can safely be removed.

Latest In Educational Video Series

The latest video comes after Google launched the educational “How Search Works” series last week to shed light on the search and indexing processes.

The newly released episode on crawling provides insight into one of the search engine’s most fundamental operations.

In the coming months, Google will produce additional episodes exploring topics like indexing, quality evaluation, and search refinements.

The series is available on the Google Search Central YouTube channel.


FAQ

What is the crawling process as described by Google?

Google’s crawling process, as outlined in their recent “How Search Works” series episode, involves the following key steps:

  • Googlebot discovers new URLs by following links from known pages it has previously crawled.
  • It strategically crawls sites at a customized speed to avoid overloading servers, taking into account response times and content quality.
  • The crawler also renders pages using the latest version of Chrome to display content loaded by JavaScript correctly and only access publicly available pages.
  • Optimizing technical SEO factors and utilizing sitemaps can facilitate Google’s crawling of new content.

How can marketers ensure their content is effectively discovered and crawled by Googlebot?

Marketers can adopt the following strategies to enhance their content’s discoverability and crawlability for Googlebot:

  • Implement an automated sitemap generation within their content management systems.
  • Focus on optimizing technical SEO elements such as site architecture and load speed and appropriately use crawl directives.
  • Ensure frequent content updates do not exhaust the crawl budget by configuring the CMS efficiently and using pagination tags.
  • Create an effective internal linking structure that helps discover new URLs.
  • Check and optimize the website’s robots.txt file to ensure it is not overly restrictive to Googlebot.