Google Cautions On Blocking GoogleOther Bot via @sejournal, @martinibuster

Google’s Gary Illyes answered a question about the non-search features that the GoogleOther crawler supports, then added a caution about the consequences of blocking GoogleOther.

What Is GoogleOther?

GoogleOther is a generic crawler created by Google for the various purposes that fall outside of those of bots that specialize for Search, Ads, Video, Images, News, Desktop and Mobile. It can be used by internal teams at Google for research and development in relation to various products.

The official description of GoogleOther is:

“GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.”

Something that may be surprising is that there are actually three kinds of GoogleOther crawlers.

Three Kinds Of GoogleOther Crawlers

  1. GoogleOther
    Generic crawler for public URLs
  2. GoogleOther-Image
    Optimized to crawl public image URLs
  3. GoogleOther-Video
    Optimized to crawl public video URLs

All three GoogleOther crawlers can be used for research and development purposes. That’s just one purpose that Google publicly acknowledges that all three versions of GoogleOther could be used for.

What Non-Search Features Does GoogleOther Support?

Google doesn’t say what specific non-search features GoogleOther supports, probably because it doesn’t really “support” a specific feature. It exists for research and development crawling which could be in support of a new product or an improvement in a current product, it’s a highly open and generic purpose.

This is the question asked that Gary narrated:

“What non-search features does GoogleOther crawling support?”

Gary Illyes answered:

“This is a very topical question, and I think it is a very good question. Besides what’s in the public I don’t have more to share.

GoogleOther is the generic crawler that may be used by various product teams for fetching publicly accessible content from sites. For example, it may be used for one-off crawls for internal research and development.

Historically Googlebot was used for this, but that kind of makes things murky and less transparent, so we launched GoogleOther so you have better controls over what your site is crawled for.

That said GoogleOther is not tied to a single product, so opting out of GoogleOther crawling might affect a wide range of things across the Google universe; alas, not Search, search is only Googlebot.”

It Might Affect A Wide Range Of Things

Gary is clear that blocking GoogleOther wouldn’t have an affect on Google Search because Googlebot is the crawler used for indexing content. So if blocking any of the three versions of GoogleOther is something a site owner wants to do, then it should be okay to do that without a negative effect on search rankings.

But Gary also cautioned about the outcome that blocking GoogleOther, saying that it would have an effect on other products and services across Google. He didn’t state which other products it could affect nor did he elaborate on the pros or cons of blocking GoogleOther.

Pros And Cons Of Blocking GoogleOther

Whether or not to block GoogleOther doesn’t necessarily have a straightforward answer. There are several considerations to whether doing that makes sense.

Pros

Inclusion in research for a future Google product that’s related to search (maps, shopping, images, a new feature in search) could be useful. It might be helpful to have a site included in that kind of research because it might be used for testing something good for a site and be one of the few sites chosen to test a feature that could increase earnings for a site.

Another consideration is that blocking GoogleOther to save on server resources is not necessarily a valid reason because GoogleOther doesn’t seem to crawl so often that it makes a noticeable impact.

If blocking Google from using site content for AI is a concern then blocking GoogleOther will have no impact on that at all. GoogleOther has nothing to do with crawling for Google Gemini apps or Vertex AI, including any future products that will be used for training associated language models. The bot for that specific use case is Google-Extended.

Cons

On the other hand it might not be helpful to allow GoogleOther if it’s being used to test something related to fighting spam and there’s something the site has to hide.

It’s possible that a site owner might not want to participate if GoogleOther comes crawling for market research or for training machine learning models (for internal purposes) that are unrelated to public-facing products like Gemini and Vertex.

Allowing GoogleOther to crawl a site for unknown purposes is like giving Google a blank check to use your site data in any way they see fit outside of training public-facing LLMs or purposes related to named bots like GoogleBot.

Takeaway

Should you block GoogleOther? It’s a coin toss. There are possible potential benefits but in general there isn’t enough information to make an informed decision.

Listen to the Google SEO Office Hours podcast at the 1:30 minute mark:

Featured Image by Shutterstock/Cast Of Thousands

What Can AI Do For Healthcare Marketing In 2024? via @sejournal, @CallRail

This post was sponsored by CallRail. The opinions expressed in this article are the sponsor’s own.

Artificial intelligence (AI) has huge potential for healthcare practices. It can assist with diagnosis and treatment, as well as administrative and marketing tasks. Yet, many practices are still wary of using AI, especially regarding marketing.

The reality is that AI is here to stay, and many healthcare practices are beginning to use the technology. According to one recent study, 89% of healthcare professionals surveyed said that they were at least evaluating AI products, experimenting with them, or had implemented AI.

To help you determine whether using AI is right for your healthcare practice, let’s take a look at some of the pros and cons of using AI while marketing.

The Pros And Cons Of AI For Healthcare Practices

Healthcare practices that choose to implement AI in safe and appropriate ways to help them with their marketing and patient experience efforts can reap many benefits, including more leads, conversions, and satisfied patients. In fact, 41% of healthcare organizations say their marketing team already uses AI.

Patients also expect healthcare practices to begin to implement AI in a number of ways. In one dentistry study, patients overall showed a positive attitude toward using AI. So, what’s holding your practice back from adding new tools and finding new use cases for AI? Let’s take a look at common concerns.

Con #1: Data Security And Privacy Concerns

Let’s get one of the biggest concerns with AI and healthcare out of the way first. Healthcare practices must follow all privacy and security regulations related to patients’ protected health information (PHI) to maintain HIPAA compliance.

So, concerns over whether AI can be used in a way that doesn’t interfere with HIPAA compliance are valid. In addition, there are also concerns about the open-source nature of popular GenAI models, which means sensitive practice data might be exposed to competitors or even hackers.

Pro #1: AI Can Help You Get More Value From Your Data Securely

While there are valid concerns about how AI algorithms make decisions and data privacy concerns, AI can also be used to enrich data to help you achieve your marketing goals while still keeping it protected.

With appropriate guardrails and omission procedures in place, you can apply AI to gain insights from data that matters to you without putting sensitive data at risk.

For example, our CallRail Labs team is helping marketers remove their blind spots by using AI to analyze and detect critical context clues that help you qualify which calls are your best leads so you can follow up promptly.

At the same time, we know how important it is for healthcare companies to keep PHI secure, which is why we integrate with healthcare privacy platforms like Freshpaint. It can help you bridge the gap between patient privacy and digital marketing.

In addition, our AI-powered Healthcare Plan automatically redacts sensitive patient-protected health information from call transcripts, enforces obligatory log-outs to prevent PHI from becoming public, provides full audit trail logging, and even features unique logins and credentials for every user, which helps eliminate the potential for PHI to be accidentally exposed to employees who don’t need access to that information.

Con #2: AI Is Impersonal

Having a good patient experience is important to almost all patients, and according to one survey, 52% of patients said a key part of a good patient experience is being treated with respect. Almost as many (46%) said they want to be addressed as a person. Given these concerns, handing over content creation or customer interactions to AI can feel daunting. While an AI-powered chatbot might be more efficient than a human in a call center, you also don’t want patients to feel like you’ve delegated customer service to a robot. Trust is the key to building patient relationships.

Pro #2: AI Can Improve The Patient Experience

Worries over AI making patient interactions feel impersonal are reasonable, but just like any other type of tool, it’s how you use AI that matters. There are ways to deploy AI that can actually enhance the patient experience and, by doing so, give your healthcare practice an advantage over your competitors.

The answer isn’t in offloading customer interaction to chatbots. But AI can help you analyze customer interactions to make customer service more efficient and helpful.

With CallRail’s AI-powered Premium Conversation Intelligence™, which transcribes, summarizes, and analyzes each call, you can quickly assess your patients’ needs and concerns and respond appropriately with a human touch. For instance, Premium Conversation Intelligence can identify and extract common keywords and topics from call transcripts. This data reveals recurring themes, such as frequently asked questions, common complaints, and popular services. A healthcare practice could then use these insights to tailor their marketing campaigns to address the most pressing patient concerns.

Con #3: AI Seems Too Complicated To Use

Let’s face it: new technology is risky, and for healthcare practices especially, risk is scary. With AI, some of the risk comes from its perceived complexity. Identifying the right use cases for your practice, selecting the right tools, training your staff, and changing workflows can all feel quite daunting. Figuring this out takes time and money. And, if there aren’t clear use cases and ROI attached, the long-term benefits may not be worth the short-term impact on business.

Pro #3: AI Can Save Time And Money

Using a computer or a spreadsheet for the first time probably also felt complicated – and on the front end, took some time to learn. However, you know that using these tools, compared to pen, paper, and calculators, has saved an enormous amount of time, making the upfront investment clearly worth it. Compared to many technologies, AI tools are often intuitive and only require you to learn a few simple things like writing prompts, refining prompts, reviewing reports, etc. Even if it takes some time to learn new AI tools, the time savings will be worth it once you do.

To get the greatest return on investment, focus on AI solutions that take care of time-intensive tasks to free up time for innovation. With the right use cases and tools, AI can help solve complexity without adding complexity. For example, with Premium Conversation Intelligence, our customers spend 60% less time analyzing calls each week, and they’re using that time to train staff better, increase their productivity, and improve the patient experience.

Con #4: AI Marketing Can Hurt Your Brand

Many healthcare practices are excited to use GenAI tools to accelerate creative marketing efforts, like social media image creation and article writing. But consumers are less excited. In fact, consumers are more likely to say that the use of AI makes them distrusting (40%), rather than trusting (19%), of a brand. In a market where trust is the most important factor for patients when choosing healthcare providers, there is caution and hesitancy around using GenAI for marketing.

Pro #4: AI Helps Make Your Marketing Better

While off-brand AI images shared on social media can be bad brand marketing, there are many ways AI can elevate your marketing efforts without impacting the brand perception. From uncovering insights to improving your marketing campaigns and maximizing the value of each marketing dollar spent to increasing lead conversion rates and decreasing patient churn, AI can help you tackle these problems faster and better than ever.

At CallRail, we’re using AI to tackle complex challenges like multi-conversation insights. CallRail can give marketers instant access to a 3-6 sentence summary for each call, average call sentiment, notable trends behind positive and negative interactions, and a summary of commonly asked questions. Such analysis would take hours and hours for your marketing team to do manually, but with AI, you have call insights at your fingertips to help drive messaging and keyword decisions that can improve your marketing attribution and the patient experience.

Con #5: Adapting AI Tools Might Cause Disruption

As a modern healthcare practice, your tech stack is the engine that runs your business. When onboarding any new technology, there are always concerns about how well it will integrate with existing technology and tools you use and whether it supports HIPAA compliance. There may also be concern about how AI tools can fit into your existing workflows without causing disruption.

Pro #5: AI Helps People Do Their Jobs Better

Pairing the right AI tool for roles with repetitive tasks can be a win for your staff and your practice. For example, keeping up with healthcare trends is important for marketers to improve messaging and campaigns.

An AI-powered tool that analyzes conversations and provides call highlights can help healthcare marketers identify keyword and Google Ad opportunities so they can focus on implementing the most successful marketing strategy rather than listening to hours of call recordings. In addition, CallRail’s new AI-powered Convert Assist helps healthcare marketers provide a better patient experience. With AI-generated call coaching, marketers can identify what went well and what to improve after every conversation.

What’s more, with a solution like CallRail, which offers a Healthcare Plan and will sign a business associate agreement (BAA), you are assured that we will comply with HIPAA controls within our service offerings to ensure that your call tracking doesn’t expose you to potential fines or litigation. Moreover, we also integrate with other marketing tools, like Google Ads, GA4, and more, making it easy to integrate our solution into your existing technologies and workflows.

Let CallRail Show You The Pros Of AI

If you’re still worried about using AI in your healthcare practice, start with a trusted solution like CallRail that has proven ROI for AI-powered tools and a commitment to responsible AI development. You can talk to CallRail’s experts or test the product out for yourself with a 14-day free trial.


Image Credits

Featured Image: Image by CallRail. Used with permission.

Find Keyword Cannibalization Using OpenAI’s Text Embeddings With Examples via @sejournal, @vahandev

This new series of articles focuses on working with LLMs to scale your SEO tasks. We hope to help you integrate AI into SEO so you can level up your skills.

We hope you enjoyed the previous article and understand what vectors, vector distance, and text embeddings are.

Following this, it’s time to flex your “AI knowledge muscles” by learning how to use text embeddings to find keyword cannibalization.

We will start with OpenAI’s text embeddings and compare them.

Model Dimensionality Pricing Notes
text-embedding-ada-002 1536 $0.10 per 1M tokens Great for most use cases.
text-embedding-3-small 1536 $0.002 per 1M tokens Faster and cheaper but less accurate
text-embedding-3-large 3072 $0.13 per 1M tokens More accurate for complex long text-related tasks, slower

(*tokens can be considered as words words.)

But before we start, you need to install Python and Jupyter on your computer.

Jupyter is a web-based tool for professionals and researchers. It allows you to perform complex data analysis and machine learning model development using any programming language.

Don’t worry – it’s really easy and takes little time to finish the installations. And remember, ChatGPT is your friend when it comes to programming.

In a nutshell:

  • Download and install Python.
  • Open your Windows command line or terminal on Mac.
  • Type this commands pip install jupyterlab and pip install notebook
  • Run Jupiter by this command: jupyter lab

We will use Jupyter to experiment with text embeddings; you’ll see how fun it is to work with!

But before we start, you must sign up for OpenAI’s API and set up billing by filling your balance.

Open AI Api Billing settingsOpen AI Api Billing settings

Once you’ve done that, set up email notifications to inform you when your spending exceeds a certain amount under Usage limits.

Then, obtain API keys under Dashboard > API keys, which you should keep private and never share publicly.

OpenAI API keysOpenAI API keys

Now, you have all the necessary tools to start playing with embeddings.

  • Open your computer command terminal and type jupyter lab.
  • You should see something like the below image pop up in your browser.
  • Click on Python 3 under Notebook.
jupyter labjupyter lab

In the opened window, you will write your code.

As a small task, let’s group similar URLs from a CSV. The sample CSV has two columns: URL and Title. Our script’s task will be to group URLs with similar semantic meanings based on the title so we can consolidate those pages into one and fix keyword cannibalization issues.

Here are the steps you need to do:

Install required Python libraries with the following commands in your PC’s terminal (or in Jupyter notebook)

pip install pandas openai scikit-learn numpy unidecode

The ‘openai’ library is required to interact with the OpenAI API to get embeddings, and ‘pandas’ is used for data manipulation and handling CSV file operations.

The ‘scikit-learn’ library is necessary for calculating cosine similarity, and ‘numpy’ is essential for numerical operations and handling arrays. Lastly, unidecode is used to clean text.

Then, download the sample sheet as a CSV, rename the file to pages.csv, and upload it to your Jupyter folder where your script is located.

Set your OpenAI API key to the key you obtained in the step above, and copy-paste the code below into the notebook.

Run the code by clicking the play triangle icon at the top of the notebook.


import pandas as pd
import openai
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
import csv
from unidecode import unidecode

# Function to clean text
def clean_text(text: str) -> str:
    # First, replace known problematic characters with their correct equivalents
    replacements = {
        '–': '–',   # en dash
        '’': '’',   # right single quotation mark
        '“': '“',   # left double quotation mark
        '”': '”',   # right double quotation mark
        '‘': '‘',   # left single quotation mark
        'â€': '—'     # em dash
    }
    for old, new in replacements.items():
        text = text.replace(old, new)
    # Then, use unidecode to transliterate any remaining problematic Unicode characters
    text = unidecode(text)
    return text

# Load the CSV file with UTF-8 encoding from root folder of Jupiter project folder
df = pd.read_csv('pages.csv', encoding='utf-8')

# Clean the 'Title' column to remove unwanted symbols
df['Title'] = df['Title'].apply(clean_text)

# Set your OpenAI API key
openai.api_key = 'your-api-key-goes-here'

# Function to get embeddings
def get_embedding(text):
    response = openai.Embedding.create(input=[text], engine="text-embedding-ada-002")
    return response['data'][0]['embedding']

# Generate embeddings for all titles
df['embedding'] = df['Title'].apply(get_embedding)

# Create a matrix of embeddings
embedding_matrix = np.vstack(df['embedding'].values)

# Compute cosine similarity matrix
similarity_matrix = cosine_similarity(embedding_matrix)

# Define similarity threshold
similarity_threshold = 0.9  # since threshold is 0.1 for dissimilarity

# Create a list to store groups
groups = []

# Keep track of visited indices
visited = set()

# Group similar titles based on the similarity matrix
for i in range(len(similarity_matrix)):
    if i not in visited:
        # Find all similar titles
        similar_indices = np.where(similarity_matrix[i] >= similarity_threshold)[0]
        
        # Log comparisons
        print(f"nChecking similarity for '{df.iloc[i]['Title']}' (Index {i}):")
        print("-" * 50)
        for j in range(len(similarity_matrix)):
            if i != j:  # Ensure that a title is not compared with itself
                similarity_value = similarity_matrix[i, j]
                comparison_result = 'greater' if similarity_value >= similarity_threshold else 'less'
                print(f"Compared with '{df.iloc[j]['Title']}' (Index {j}): similarity = {similarity_value:.4f} ({comparison_result} than threshold)")

        # Add these indices to visited
        visited.update(similar_indices)
        # Add the group to the list
        group = df.iloc[similar_indices][['URL', 'Title']].to_dict('records')
        groups.append(group)
        print(f"nFormed Group {len(groups)}:")
        for item in group:
            print(f"  - URL: {item['URL']}, Title: {item['Title']}")

# Check if groups were created
if not groups:
    print("No groups were created.")

# Define the output CSV file
output_file = 'grouped_pages.csv'

# Write the results to the CSV file with UTF-8 encoding
with open(output_file, 'w', newline='', encoding='utf-8') as csvfile:
    fieldnames = ['Group', 'URL', 'Title']
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    
    writer.writeheader()
    for group_index, group in enumerate(groups, start=1):
        for page in group:
            cleaned_title = clean_text(page['Title'])  # Ensure no unwanted symbols in the output
            writer.writerow({'Group': group_index, 'URL': page['URL'], 'Title': cleaned_title})
            print(f"Writing Group {group_index}, URL: {page['URL']}, Title: {cleaned_title}")

print(f"Output written to {output_file}")

This code reads a CSV file, ‘pages.csv,’ containing titles and URLs, which you can easily export from your CMS or get by crawling a client website using Screaming Frog.

Then, it cleans the titles from non-UTF characters, generates embedding vectors for each title using OpenAI’s API, calculates the similarity between the titles, groups similar titles together, and writes the grouped results to a new CSV file, ‘grouped_pages.csv.’

In the keyword cannibalization task, we use a similarity threshold of 0.9, which means if cosine similarity is less than 0.9, we will consider articles as different. To visualize this in a simplified two-dimensional space, it will appear as two vectors with an angle of approximately 25 degrees between them.

<span class=

In your case, you may want to use a different threshold, like 0.85 (approximately 31 degrees between them), and run it on a sample of your data to evaluate the results and the overall quality of matches. If it is unsatisfactory, you can increase the threshold to make it more strict for better precision.

You can install ‘matplotlib’ via terminal.

pip install matplotlib

And use the Python code below in a separate Jupyter notebook to visualize cosine similarities in two-dimensional space on your own. Try it; it’s fun!


import matplotlib.pyplot as plt
import numpy as np

# Define the angle for cosine similarity of 0.9. Change here to your desired value. 
theta = np.arccos(0.9)

# Define the vectors
u = np.array([1, 0])
v = np.array([np.cos(theta), np.sin(theta)])

# Define the 45 degree rotation matrix
rotation_matrix = np.array([
    [np.cos(np.pi/4), -np.sin(np.pi/4)],
    [np.sin(np.pi/4), np.cos(np.pi/4)]
])

# Apply the rotation to both vectors
u_rotated = np.dot(rotation_matrix, u)
v_rotated = np.dot(rotation_matrix, v)

# Plotting the vectors
plt.figure()
plt.quiver(0, 0, u_rotated[0], u_rotated[1], angles='xy', scale_units='xy', scale=1, color='r')
plt.quiver(0, 0, v_rotated[0], v_rotated[1], angles='xy', scale_units='xy', scale=1, color='b')

# Setting the plot limits to only positive ranges
plt.xlim(0, 1.5)
plt.ylim(0, 1.5)

# Adding labels and grid
plt.xlabel('X-axis')
plt.ylabel('Y-axis')
plt.grid(True)
plt.title('Visualization of Vectors with Cosine Similarity of 0.9')

# Show the plot
plt.show()

I usually use 0.9 and higher for identifying keyword cannibalization issues, but you may need to set it to 0.5 when dealing with old article redirects, as old articles may not have nearly identical articles that are fresher but partially close.

It may also be better to have the meta description concatenated with the title in case of redirects, in addition to the title.

So, it depends on the task you are performing. We will review how to implement redirects in a separate article later in this series.

Now, let’s review the results with the three models mentioned above and see how they were able to identify close articles from our data sample from Search Engine Journal’s articles.

Data SampleData Sample

From the list, we already see that the 2nd and 4th articles cover the same topic on ‘meta tags.’ The articles in the 5th and 7th rows are pretty much the same – discussing the importance of H1 tags in SEO – and can be merged.

The article in the 3rd row doesn’t have any similarities with any of the articles in the list but has common words like “Tag” or “SEO.”

The article in the 6th row is again about H1, but not exactly the same as H1’s importance to SEO. Instead, it represents Google’s opinion on whether they should match.

Articles on the 8th and 9th rows are quite close but still different; they can be combined.

text-embedding-ada-002

By using ‘text-embedding-ada-002,’ we precisely found the 2nd and 4th articles with a cosine similarity of 0.92 and the 5th and 7th articles with a similarity of 0.91.

Screenshot from Jupyter log showing cosine similaritiesScreenshot from Jupyter log showing cosine similarities

And it generated output with grouped URLs by using the same group number for similar articles. (colors are applied manually for visualization purposes).

Output sheet with grouped URLsOutput sheet with grouped URLs

For the 2nd and 3rd articles, which have common words “Tag” and “SEO” but are unrelated, the cosine similarity was 0.86. This shows why a high similarity threshold of 0.9 or greater is necessary. If we set it to 0.85, it would be full of false positives and could suggest merging unrelated articles.

text-embedding-3-small

By using ‘text-embedding-3-small,’ quite surprisingly, it didn’t find any matches per our similarity threshold of 0.9 or higher.

For the 2nd and 4th articles, cosine similarity was 0.76, and for the 5th and 7th articles, with similarity 0.77.

To better understand this model through experimentation, I’ve added a slightly modified version of the 1st row with ’15’ vs. ’14’ to the sample.

  1. “14 Most Important Meta And HTML Tags You Need To Know For SEO”
  2. “15 Most Important Meta And HTML Tags You Need To Know For SEO”
Example which shows text-embedding-3-small resultsAn example which shows text-embedding-3-small results

On the contrary, ‘text-embedding-ada-002’ gave 0.98 cosine similarity between those versions.

Title 1 Title 2 Cosine Similarity
14 Most Important Meta And HTML Tags You Need To Know For SEO 15 Most Important Meta And HTML Tags You Need To Know For SEO 0.92
14 Most Important Meta And HTML Tags You Need To Know For SEO Meta Tags: What You Need To Know For SEO 0.76

Here, we see that this model is not quite a good fit for comparing titles.

text-embedding-3-large

This model’s dimensionality is 3072, which is 2 times higher than that of ‘text-embedding-3-small’ and ‘text-embedding-ada-002′, with 1536 dimensionality.

As it has more dimensions than the other models, we could expect it to capture semantic meaning with higher precision.

However, it gave the 2nd and 4th articles cosine similarity of 0.70 and the 5th and 7th articles similarity of 0.75.

I’ve tested it again with slightly modified versions of the first article with ’15’ vs. ’14’ and without ‘Most Important’ in the title.

  1. “14 Most Important Meta And HTML Tags You Need To Know For SEO”
  2. “15 Most Important Meta And HTML Tags You Need To Know For SEO”
  3. “14 Meta And HTML Tags You Need To Know For SEO”
Title 1 Title 2 Cosine Similarity
14 Most Important Meta And HTML Tags You Need To Know For SEO 15 Most Important Meta And HTML Tags You Need To Know For SEO 0.95
14 Most Important Meta And HTML Tags You Need To Know For SEO 14 Most Important Meta And HTML Tags You Need To Know For SEO 0.93
14 Most Important Meta And HTML Tags You Need To Know For SEO Meta Tags: What You Need To Know For SEO 0.70
15 Most Important Meta And HTML Tags You Need To Know For SEO 14 Most Important  Meta And HTML Tags You Need To Know For SEO 0.86

So we can see that ‘text-embedding-3-large’ is underperforming compared to ‘text-embedding-ada-002’ when we calculate cosine similarities between titles.

I want to note that the accuracy of ‘text-embedding-3-large’ increases with the length of the text, but ‘text-embedding-ada-002’ still performs better overall.

Another approach could be to strip away stop words from the text. Removing these can sometimes help focus the embeddings on more meaningful words, potentially improving the accuracy of tasks like similarity calculations.

The best way to determine whether removing stop words improves accuracy for your specific task and dataset is to empirically test both approaches and compare the results.

Conclusion

With these examples, you have learned how to work with OpenAI’s embedding models and can already perform a wide range of tasks.

For similarity thresholds, you need to experiment with your own datasets and see which thresholds make sense for your specific task by running it on smaller samples of data and performing a human review of the output.

Please note that the code we have in this article is not optimal for large datasets since you need to create text embeddings of articles every time there is a change in your dataset to evaluate against other rows.

To make it efficient, we must use vector databases and store embedding information there once generated. We will cover how to use vector databases very soon and change the code sample here to use a vector database.

More resources: 


Featured Image: BestForBest/Shutterstock

Study Backs Google’s Claims: AI Search Boosts User Satisfaction via @sejournal, @MattGSouthern

A new study finds that despite concerns about AI in online services, users are more satisfied with search engines and social media platforms than before.

The American Customer Satisfaction Index (ACSI) conducted its annual survey of search and social media users, finding that satisfaction has either held steady or improved.

This comes at a time when major tech companies are heavily investing in AI to enhance their services.

Search Engine Satisfaction Holds Strong

Google, Bing, and other search engines have rapidly integrated AI features into their platforms over the past year. While critics have raised concerns about potential negative impacts, the ACSI study suggests users are responding positively.

Google maintains its position as the most satisfying search engine with an ACSI score of 81, up 1% from last year. Users particularly appreciate its AI-powered features.

Interestingly, Bing and Yahoo! have seen notable improvements in user satisfaction, notching 3% gains to reach scores of 77 and 76, respectively. These are their highest ACSI scores in over a decade, likely due to their AI enhancements launched in 2023.

The study hints at the potential of new AI-enabled search functionality to drive further improvements in the customer experience. Bing has seen its market share improve by small but notable margins, rising from 6.35% in the first quarter of 2023 to 7.87% in Q1 2024.

Customer Experience Improvements

The ACSI study shows improvements across nearly all benchmarks of the customer experience for search engines. Notable areas of improvement include:

  • Ease of navigation
  • Ease of using the site on different devices
  • Loading speed performance and reliability
  • Variety of services and information
  • Freshness of content

These improvements suggest that AI enhancements positively impact various aspects of the search experience.

Social Media Sees Modest Gains

For the third year in a row, user satisfaction with social media platforms is on the rise, increasing 1% to an ACSI score of 74.

TikTok has emerged as the new industry leader among major sites, edging past YouTube with a score of 78. This underscores the platform’s effective use of AI-driven content recommendations.

Meta’s Facebook and Instagram have also seen significant improvements in user satisfaction, showing 3-point gains. While Facebook remains near the bottom of the industry at 69, Instagram’s score of 76 puts it within striking distance of the leaders.

Challenges Remain

Despite improvements, the study highlights ongoing privacy and advertising challenges for search engines and social media platforms. Privacy ratings for search engines remain relatively low but steady at 79, while social media platforms score even lower at 73.

Advertising experiences emerge as a key differentiator between higher- and lower-satisfaction brands, particularly in social media. New ACSI benchmarks reveal user concerns about advertising content’s trustworthiness and personal relevance.

Why This Matters For SEO Professionals

This study provides an independent perspective on how users are responding to the AI push in online services. For SEO professionals, these findings suggest that:

  1. AI-enhanced search features resonate with users, potentially changing search behavior and expectations.
  2. The improving satisfaction with alternative search engines like Bing may lead to a more diverse search landscape.
  3. The continued importance of factors like content freshness and site performance in user satisfaction aligns with long-standing SEO best practices.

As AI becomes more integrated into our online experiences, SEO strategies may need to adapt to changing user preferences.


Featured Image: kate3155/Shutterstock

OpenAI Launches SearchGPT: AI-Powered Search Prototype via @sejournal, @MattGSouthern

OpenAI has announced the launch of SearchGPT, a prototype AI-powered search engine.

This move marks the company’s entry into the competitive search market, potentially challenging established players.

Key Features & Functionality

SearchGPT aims to directly answer user queries by combining AI language models with real-time web information.

Rather than offering a list of links, SearchGPT attempts to deliver concise responses with citations to source material.

Here’s an example of a search results page for the query: “music festivals in boone north carolina in august.”

Screenshot from openai.com/index/searchgpt-prototype/, July 2024.

The SearchGPT prototype includes:

  • A conversational interface allowing follow-up questions
  • Real-time information retrieval from web sources
  • In-line attributions and links to original content

Publisher Controls & Content Management

OpenAI is also introducing tools for publishers to manage how their content appears in SearchGPT, giving them more control over their presence in AI-powered search results.

Key points about the publisher controls include:

  1. Separate from AI training: OpenAI emphasizes that SearchGPT is distinct from the training of their generative AI models. Sites can appear in search results even if they opt out of AI training data.
  2. Content management options: Publishers can influence how their content is displayed and used within SearchGPT.
  3. Feedback mechanism: OpenAI has provided an email (publishers-feedback@openai.com) for publishers to share their thoughts and concerns.
  4. Performance insights: The company plans to share information with publishers about their content’s performance within the AI search ecosystem.

These tools are OpenAI’s response to ongoing debates about AI’s use of web content and concerns over intellectual property rights.

Publisher Partnerships & Reactions

OpenAI reports collaborating with several publishers during the development of SearchGPT.

Nicholas Thompson, CEO of The Atlantic, provided a statement supporting the initiative, emphasizing the importance of valuing and protecting journalism in AI search development.

Robert Thomson, News Corp’s chief executive, also commented on the project, stressing the need for a symbiotic relationship between technology and content and the importance of protecting content provenance.

Limited Availability & Future Plans

Currently, SearchGPT is available to a restricted group of users and publishers.

OpenAI describes it as a temporary prototype, indicating plans to integrate features into their existing ChatGPT product eventually.

Why This Matters

The introduction of SearchGPT represents a potential shakeup to the search engine market.

This development could have far-reaching implications for digital marketing, content creation, and user behavior on the internet.

Potential effects include:

  • Changes in content distribution and discovery mechanisms
  • New considerations for search engine optimization strategies
  • Evolving relationships between AI companies and content creators

Remember, this is still a prototype, and we have yet to see its capabilities.

There’s a waitlist available for those trying to get their hands on it early.

What This Means For You

AI-powered search might offer users more direct access to information. However, the accuracy and comprehensiveness of results may depend on publisher participation and content management choices.

For content creators and publishers, these new tools provide opportunities to have more say in how their work is used in AI search contexts.

While it may increase content visibility and engagement, it also requires adapting to new formats and strategies to ensure content is AI-friendly and easily discoverable.

As SearchGPT moves from prototype to integration with ChatGPT, it will be vital to stay informed about these developments and adapt your strategies.

The future of search is evolving, and AI is at the forefront of this transformation.

Agile SEO: Moving From Strategy To Action via @sejournal, @jes_scholz

Impactful SEO is rarely executed by a lone wolf.

You need resources. You need buy-in from higher-ups – a CMO, head of product, or even CEO.

But here’s the thing: those lengthy SEO documents outlining objectives, audiences, competitors, keywords, and that six-month Gantt chart vaguely detailing optimization projects – they’re not getting read.

On the contrary, it is a roadblock to you getting a green light for resources.

An executive can quickly scan one short email containing a clear request and sign off. However, they need to set aside dedicated time to read a strategy document in depth – and time is not something executives have a lot of.

And even if they sign off today, the reality is business priorities shift. Competitive landscapes change. Algorithms are updated.

SEO is executed in a constant state of flux. It demands flexibility on a monthly, even weekly basis.

So, let’s ditch the long documents and prioritize actions over proposals with agile SEO.

Why Agile SEO Strategies Work

Agile SEO involves incremental iteration.

Break complex, overarching projects down into small, frequent changes.

Enable continual progress.

google quoteImage from author

Forget the pursuit of SEO perfection.

The key is to launch a minimum viable product (MVP) and monitor the impact on metrics.

Once you are armed with performance data, you can move on. The key performance indicator (KPI) impact will get you buy-in for the resources you need.

Let me give you an example.

Say your overarching goal is to completely overhaul the website architecture of an e-commerce site – all the URL routes, page titles, meta descriptions, and H1s for the homepage, category pages, and product pages.

The Old Way: One Giant Leap

The traditional approach involves pitching the entire SEO project at once. Your argument is that it’s good for SEO.

The site will rank higher and significantly impact the organic sessions. Which is true.

However, the document communicating all the reasons and requirements is complicated to review.

The project will seem too large. It will likely not make it onto your development team’s roadmap, as they will likely feel your request will overload their development cycle.

overloaded donkeyImage from author

Agile SEO Approach: Small Iterations

What if you broke it down into micro-wins?

Instead of pitching the entire project, request approval for a small but impactful change.

For example, optimizing the title tag and meta description of the homepage.

The documentation for this will be less than one page. The change request is equivalent to snackable content. Because it’s easy to implement, it’s much easier to incorporate it into a development sprint.

Now, say this quick change positively impacts KPIs, such as a 3% lift in homepage organic sessions. You can then argue for similar changes for the category pages, pointing out that if we get a similar KPI lift as we did for the homepage, this will achieve X more organic sessions.

You have already proven such tactics can increase KPIs. So, there is more trust in your approach. And it’s, again, a small request. So, your development team is more likely to do it.

And you can rinse and repeat until you have the whole site migrated.

How To Document An Agile SEO Strategy

So now we know to stop writing long SEO strategy documents and instead start creating agile, “snackable” tactics.

But we still need to understand what:

  • Has been completed in the past.
  • Is being worked on now.
  • Is coming up next.
  • All the ideas are.

This information must be easy to digest, centrally accessible, and flexible.

One solution for this is an “SEO calendar” document.

<span class=

Elements of an SEO calendar:

  • Date column: Ideally matched against IT sprint cycles. This does not mean every SEO initiative involves IT. But if you need a developer’s assistance, it will simplify cross-functional team projects. Having it set, for example, every two weeks also promotes small but constant releases from the SEO team.
  • Backlog: This provides space for team members to record ideas without having to make any significant commitment of time. Assess all ideas regularly as you fill your next available calendar slot.
  • Change column: A clear and concise sentence on what has been or will be changed.
  • Tactic brief: A link to the detailed information of that test. More details coming below.
  • Sign off: Ensuring all SEO changes pass a four-eye principle from a strategic point of view lowers the risk of any errors. These quick-to-read, snackable briefs make it easy to get your managers to buy in and sign off for resources.
  • Outcome: One short sentence summing up the KPI impact.

The benefit of a calendar layout is it is fully flexible but time-relevant. Changing priorities is as simple as moving the de-prioritized item to the backlog.

It can act as a website change log for SEO. Everyone can know the timetable of changes, both past and planned upcoming.

Those interested in why the KPIs increased on a certain date have the answer at a glance and more detailed information in one click. This can be invaluable for troubleshooting.

And, for team leaders, if any gaps appear in the iteration cycle, you can see this as gaps will appear in the calendar, allowing you to address the root cause.

Snackable Tactic Briefs

The benefits of tactics briefs are twofold:

  • Pre-launch: They concisely answer the Five Ws of your SEO change to get buy-in from stakeholders. Once aligned, it will act as the specification if you need someone else to execute it.
  • Post-launch: Be the record of what was actually changed. What impact did it have on the KPI funnel? What did we learn? And what are the next steps, if any?

Tactics briefs have five sections:

  • Overview.
  • SMART Goal.
  • Specifications.
  • Results.
  • Learnings & Action Items.

Overview

The overview section should cover the basics of the test:

  • Who is the one person ultimately responsible for leading the execution of the test?
  • When will it (pre-launch)/did it (post-launch) go live?
  • When will we (pre-launch)/did we (post-launch) assess results?
  • Who proposed the change? (It may be important to know if you need more information on the background for the test or if an action has come from senior management.)
  • Who has agreed to this execution? (This may be development, the line manager in marketing, or another key stakeholder. Allowing everyone to see who is on board.)
Overview tableScreenshot from author

SMART Goal

The SMART goal is the high-level tactical approach.

Align your goal with your stakeholders before a detailed documentation effort goes into a task. This also ensures the change is in line with business goals.

<span class=

Specifications

This section will vary based on your test. But always try to communicate the “before” and the “after.” This way, you have a clear historical record you can refer back to.

The key is to have only the details needed. Nothing more, nothing less.

You can use tables to keep it easy to scan.

For example, in the case of a title tag change, it could be as simple as a single table.

Title tag formula for category pagesScreenshot from author

The key is to avoid long paragraphs of text. Focus on clearly communicating the outcome. What was it before, and what will be it after?

Don’t explain how the task was executed.

Results

This section should contain one table to effectively communicate the percentage change between the benchmark weeks and the SEO change from a full-funnel perspective, as well as any additional tables to drill down for more insights.

An example of a table could be similar to the one below.

Category page organic KPI results tableScreenshot from author

Learnings & Action Items

Here is where you can succinctly analyze the results.

Remember, you have the data clearly available in the table above, so you don’t need to list the numbers again.

Explain what the numbers mean and what actions will be taken next.

Final Thoughts

An agile SEO system provides flexibility and visibility.

At any time, you can understand what actions are underway and what has shifted KPIs.

Forget the fantasy of the perfect SEO strategy, and focus your energy on getting sh!t done.

More resources: 


Featured Image: Andrey_Popov/Shutterstock

Bing’s Updated AI Search Will Make Site Owners Happy via @sejournal, @martinibuster

Bing is rolling out a new version of Generative Search that displays information in an intuitive way that encourages exploration but also prioritizes clicks from the search results to websites.

Microsoft introduced their new version of AI search:

“After introducing LLM-powered chat answers on Bing in February of last year, we’ve been hard at work on the ongoing revolution of search. …Today, we’re excited to share an early view of our new generative search experience which is currently shipping to a small percentage of user queries.”

New Layout

Bing’s announcement discusses new features that not only make it easy for users to find information, Bing also makes it easy for users to see the organic search results and click through and browse websites.

On the desktop view, Bing shows three panels:

  • A table of content on the left
  • AI answers in the center (with links to website sources)
  • Traditional organic search results on the right hand side
  • Even more organic search results beneath “the fold”

The table of contents that is on the right hand side is invites exploration. It has the main topic at the top, with directly related subtopics beneath it. This is so much better than a People Also Asked type of navigation because it invites the user to explore and click on an organic search result to keep on exploring.

Screenshot: Table Of Contents

This layout is the result of a conscious decision at Bing to engineer it so that that it preserves and encourages clicks to websites.

Below is a screenshot of the new generative AI search experience. What’s notable is how Bing surrounds the AI answers with organic search results.

Screenshot Of The New Bing AI Search Results

Bing makes a point to explain that they have tested the new interface to make sure that the search results will send the same amount of traffic and to avoid creating a layout that results in an increase in zero click search results.

When other search engines talk about search quality it is always from the context of user satisfaction. Bing’s announcement makes it clear that sustaining traffic to websites was an important context that guided the design of the new layout.

Below is a screenshot of a typical Bing AI search result for a query about the life span of elephants.

Note that all the areas that I bounded with blue boxes are AI answers while everything outside of the blue boxes are organic search results.

Screenshot Of Mix of AI And Organic Results

Bing's new AI search layout emphasizes organic search results

The screenshot makes it clear that there is a balance of organic search results and AI answers. In addition to those contextually relevant organic search results there are also search results on the right hand side (not shown in the above screenshot).

Microsoft’s blog post explained:

“We are continuing to look closely at how generative search impacts traffic to publishers. Early data indicates that this experience maintains the number of clicks to websites and supports a healthy web ecosystem. The generative search experience is designed with this in mind, including retaining traditional search results and increasing the number of clickable links, like the references in the results.”

Bing’s layout is a huge departure from the zero-click style of layouts seen in other search engines. Bing has purposely designed their generative AI layout to maintain clicks to websites. It cannot be overstated how ethical Bing’s approach to the web ecosystem is.

Bing Encourages Browsing And Discovery

An interesting feature of Bing’s implementation of generative AI search is that it shows the answer to the initial question first, and it also anticipates related questions. This is similar to a technique called “information gain” where an AI search assistant will rank an initial set of pages that answers a search query, but will also rank a second, third and fourth set of search results that contain additional information that a user may be interested in, information on related topics.

What Bing does differently from the Information Gain technique is that Bing displays all the different search results on a single page and then uses a table of contents on the left hand side that makes it easy for a user to click and go straight to the additional AI answers and organic search results.

Bing’s Updated AI Search Is Rolling Out Now

Bing’s newly updated AI search engine layout is slowly rolling out and they are observing the feedback from users. Microsoft has already tested it and is confident that it will continue to send clicks to websites. Search engines have a relationship with websites, what is commonly referred to as the web ecosystem. Every strong relationship is based on giving, not taking. When both sides give it creates a situation where both sides receive.

More search engines should take Bing’s approach of engineering their search results to satisfy users in a way that encourages discovery on the websites that originate the content.

Read Bing’s announcement:

Introducing Bing generative search

Featured Image by Shutterstock/Primakov

Google Search Revenue Grows 14% In Q2 2024 via @sejournal, @MattGSouthern

Alphabet Inc., Google’s parent company, released its second quarter 2024 financial results, revealing a 14% year-over-year increase in revenue for its core Google Search business.

Key Financial Data:

  • Google Search revenue: $48.5 billion (up from $42.6 billion in Q2 2023)
  • Total Alphabet revenue: $84.7 billion (14% increase year-over-year)
  • Operating income: $27.4 billion
  • Net income: $23.6 billion
  • Earnings per share: $1.89

Strong performances in Search and Cloud services primarily drove the company’s overall revenue growth.

Google Cloud surpassed $10 billion in quarterly revenue for the first time, reaching $10.3 billion with $1.2 billion in operating profit.

YouTube ad revenue increased from $7.7 billion in Q2 2023 to $8.7 billion in Q2 2024.

Alphabet CEO Sundar Pichai commented on the results, emphasizing the company’s focus on AI innovation. The report also noted a recent reorganization of AI teams, combining elements of Google Research with Google DeepMind.

Pichai stated:

“Our strong performance this quarter highlights ongoing strength in Search and momentum in Cloud. We are innovating at every layer of the AI stack.”

The report also noted a recent reorganization of AI teams, combining elements of Google Research with Google DeepMind.

While the results indicate strong performance, Alphabet faces challenges, including regulatory scrutiny and evolving competition in the tech sector.

The company’s CFO, Ruth Porat, mentioned ongoing efforts to optimize cost structures.

Regarding the company’s financial strategy, Porat stated:

“As we invest to support our highest growth opportunities, we remain committed to creating investment capacity with our ongoing work to durably re-engineer our cost base.”

Why This Matters

The performance of Google Search and Alphabet has implications for the digital marketing industry.

As the dominant search engine, Google’s revenue growth indicates continued strength in search advertising, which remains an essential channel for many businesses.

Additionally, the growth in Cloud services and YouTube advertising suggests evolving digital trends and potential opportunities for marketers.

What Does This Mean For You?

For digital marketers and SEO professionals, these are the key takeaways from Alphabet’s earnings call:

  • Search remains vital: The growth in Google Search revenue shows that SEO and search advertising remain key components of marketing strategies.
  • Cloud and AI focus: Alphabet’s emphasis on Cloud services and AI development may lead to new tools and platforms for marketers to leverage.
  • Video advertising potential: The growth in YouTube ad revenue indicates the ongoing importance of video content in digital marketing strategies.
  • Competitive landscape: While Google maintains its market position, the focus on AI development across the tech industry may lead to new challenges and opportunities in search and digital advertising.
  • Potential changes ahead: As Alphabet continues to invest in AI and reorganize its teams, marketers should stay alert for potential changes in search algorithms or new AI-driven features that could impact SEO and PPC strategies

Featured Image: sdx15/Shutterstock

Google Shares Tips To Improve SEO Through Internal Links via @sejournal, @MattGSouthern

In a new installment of its “SEO Made Easy” video series, Google provides three simple guidelines for utilizing internal linking to improve SEO.

The video, presented by Google’s Martin Splitt, offers valuable insights for improving site structure and user experience.

Strategic internal linking highlights your most valuable content, ensuring users and search engines can identify them quickly.

Additionally, internal linking can help search engines understand the relationships between pages, potentially leading to better rankings.

3 Tips For Internal Linking

Splitt emphasized three main points regarding the effective use of internal links:

  1. User Navigation: Internal links guide users through a website, helping them find related content and understand the site’s structure.
  2. Search Engine Crawling: Google’s web crawler, Googlebot, uses internal links to discover new pages and understand the relationships between different pages on a site.
  3. HTML Best Practices: Properly using HTML elements, particularly the < a> tag with an href attribute, is essential for creating effective links.

The Importance Of Meaningful Anchor Text

One of Google’s key recommendations is to use descriptive, meaningful anchor text for links.

Splitt demonstrated how clear anchor text improves user experience by allowing visitors to quickly scan a page and understand where each link will lead them.

He stated:

“Users and Bots alike prefer meaningful anchor text. Here on the left you see what that looks like each link has meaningful words as anchor text and you can easily spot what the link will take you to.”

See the examples he’s referring to in the image below:

Screenshot from: YouTube.com/GoogleSearchCentral, July 2024.

Splitt continues:

“On the right you see a page that doesn’t use meaningful anchor text and that isn’t a good user experience especially when you try to quickly scan the page and find the right link to use.”

Balancing Link Quantity

While internal linking is vital, Splitt cautioned against overdoing it.

He advises applying critical judgment when adding links and creating logical connections between related content without overwhelming the user or diluting the page’s focus.

Technical Considerations For Links

The video also touched on the technical aspects of link implementation.

Splitt discouraged using non-standard elements like spans, divs, or buttons to create links, saying if an element behaves like a link, it should be coded as one using the proper HTML structure.

Screenshot from: YouTube.com/GoogleSearchCentral, July 2024.

In Summary

These are the key takeaways from Google’s video on internal linking:

  • Internal linking is a fundamental aspect of SEO and user experience.
  • Focus on creating meaningful, descriptive anchor text for links.
  • Use internal links strategically to guide users and search engines through your site.
  • Balance the number of links to avoid overwhelming users or diluting page focus.
  • Stick to proper HTML structure when implementing links.

See the full video below:


Featured Image: Screenshot from YouTube.com/GoogleSearchCentral, July 2024. 

System Builders – How AI Changes The Work Of SEO via @sejournal, @Kevin_Indig

AI is terraforming tech. The content and SEO ecosystem is undergoing a massive structural change.

Human-written content gains value faster for LLM training than for end consumers as the pure profit licensing deals between LLM developers and publishers show.

Publishers struggle to survive from digital subscriptions but get millions that go straight to their bottom line for providing training data.

Content platforms, social networks, SaaS companies and consumer apps coat their products with AI. A few examples:

  • Spotify DJ (AI-generated playlist).
  • AI Overview (AI answers in Google Search).
  • Instagram AI personas (celebrity AI chatbots).
  • Ebay’s magical listing (turn a photo into a listing).
  • Redfin Redesign (try interior designs on real house pictures).
Google searches for chat gptImage Credit: Kevin Indig

The quality of machine-generated content (MGC) challenges human-generated content (HGC). I ran an experiment with my Twitter and LinkedIn followers: I asked them to choose which of two articles was written by a human and which by a machine – and they had to explain their answer.

Only a handful of people figured out that AI wrote both pieces. I intentionally framed the question in a leading way to see if people would challenge the setting or believe that one piece was written by a human if told so.

  • Not an isolated experiment: A survey of 1,900 Americans found that 63.5% of people can’t distinguish between AI content and human content.1
  • People seek help: Google search demand for [ai checker] has reached 100,000 in May 2024 (Glimpse).
  • Dark side: scammers use MGC to make money, as 77% of AI scam victims lost money.2
Search demand for AI checkerImage Credit: Kevin Indig

The quality level of LLMs pushes SEO work towards automating workflows and learning with AI, while writers will take content from good to great instead of zero to one.

Boost your skills with Growth Memo’s weekly expert insights. Subscribe for free!

How AI Changes The Work Of SEOImage Credit: Lyna ™

System Builders

Clients, podcasters and panel hosts often ask me what skills SEOs need to build for the AI future. For a long time, my answer was to learn, stay open-minded and gain as much practical experience with AI as possible.

Now, my answer is SEOs should learn how to build AI agents and workflows that automate tasks. AI changes the way search works but also the way SEOs work.

AI + No-code Allows SEOs To Automate Workflows

A few examples:

1/ Cannibalization

  • Old world: SEOs download search console data and create pivot tables to spot keyword cannibalization.
  • New world: SEOs build an AI workflow that sends alters, identifies true keyword cannibalization, makes content suggestions to fix the problem, and monitors the improvement.

2/ Site Crawling

  • Old world: SEOs crawl websites to find inefficiencies in internal linking, status code errors, duplicate content, etc.
  • New world: SEOs build an AI agent that regularly crawls the site and automatically suggests new internal links that are shipped after human approval, fixes broken canonical tags and excludes soft 404 errors in the robots.txt.

3/ Content Creation

  • Old world: SEOs do keyword research and write content briefs. Writers create the content.
  • New world: SEOs automate keyword research with AI and create hundreds of relevant articles as a foundation for writers to build on.

All of this is already possible today with AI workflow tools like AirOps or Apify, which chain agents and LLMs together to scrape, analyze, transform data or create content.

Moving forward, we’ll spend much more time building automated systems instead of wasting time on point analyses and catalogs of recommendations. The SEO work will be defining logic, setting rules, prompting and coding.

building automated systems Building workflows with AirOps (Image Credit: Kevin Indig)

You Can Learn (Almost) Anything With AI

I never made the time to really learn Python or R, but with the help of Chat GPT and Gemini in Colab, I can write any script with natural language prompts.

When the script doesn’t work, I can paste a screenshot into Chat GPT and describe the issue to get a solution. AI helps with Regex, Google Sheets/Excel, R, Python, etc. Nothing is off-limits.

Being able to write scripts can solve problems like data analysis, a/b testing and using APIs. As an SEO, I’m no longer dependent on engineers, data scientists or writers to perform certain tasks. I can act faster and on my own account.

I’m not the only one to figure this out. People are learning to code, write and many other skills with AI. We can learn to build AI workflows by asking AI to teach us.

Search demand for coding with AI is explodingImage Credit: Kevin Indig
Search demand for write with AI is explodingImage Credit: Kevin Indig
Search demand for learn with AI is explodingImage Credit: Kevin Indig

When you can learn almost anything, the only limit is time.

The Work Of Writers Changes

Against common belief, writers won’t be crossed out of this equation but will play the critical role of editing, directing and curating.

In any automated process, humans QA the output. Think of car assembling lines. Even though AI content leaps in quality, spot checks reduce the risk of errors. Caught issues, such as wrong facts, weird phrasing or off-brand wording, will be critical feedback to fine-tune models to improve their output.

Instead of leg work like writing drafts, writers will bring AI content from good to great. In the concept of information gain, writers will spend most of their time making a piece outstanding.

The rising quality work spans from blog content to programmatic content, where writers will add curated content when searches have a desire for human experience, such as in travel.

A mini guide to Los AngelesTripadvisor’s attraction pages feature human-curated sections. (Image Credit: Kevin Indig)

Unfair Advantage

As often with new technology, a few first-mover people and companies get exponential value until the rest catch up. My worry is that a few fast-moving companies will grab massive land with AI.

And yet, this jump in progress will allow newcomers to challenge incumbents and get a fair chance to compete on the field.

AI might be a bigger game changer for SEOs than for Google. The raw power of AI might help us overcome challenges from AI Overviews and machine learning-driven algorithm updates.

But the biggest win might be that SEOs can finally make something instead of delivering recommendations. The whole value contribution of SEOs changes because my output can drive results faster.

Survey: ChatGPT and AI Content – Can people tell the difference?

Artificial Intelligence Voice Scams on the Rise with 1 in 4 Adults Impacted


Featured Image: Paulo Bobita/Search Engine Journal