Long-Tail SEO in an AI World

In 2006, Wired magazine editor Chris Anderson famously described the availability of niche products online as the “long tail.” Search optimizers adopted the term, calling queries of three words or more “long-tail keywords.”

Optimizing for long-tail searches has multiple benefits. Consumers searching on extended keywords tend to know what they want, and longer queries typically have less keyword competition. Yet the biggest benefit could now be AI visibility: Generative AI platforms such as ChatGPT fan out using multiword queries to answer user prompts.

Long-Tail Queries

A seed term plus modifiers

Any long-tail query consists of a seed term and one or more modifiers. For example, “shoes” is a seed term, and potential modifiers are:

• “for women,”
• “red,”
• “near me,”
• “on-sale.”

Combining the seed term and modifiers — “red shoes for women,” “on sale near me” — yields narrow queries that describe searchers’ needs, such as gender, color, location, and price.

Modifiers reflect the searcher’s intent and stage in a buying journey, from exploration to purchase. Thus, keyword research is the process of extending a core term with modifiers to optimize a site for buying journeys.

The more modifiers, the more specific the intent and, typically, the lesser the volume and clicks. Conversely, more modifiers improve the likelihood of conversions, provided the content of the landing page follows closely from that phrase. A query of “red shoes for women” should link to a page with women wearing red shoes.

Types of modifiers

A core term can have many modifiers, such as:

• Location,
• Description (“red”),
• Price (typically from searchers eager to buy),
• Brand,
• Age and gender,
• Questions (“how to clean shoes”).

Long-Tail Opportunities

Keyword research tools

Grouping keywords by modifier type can reveal your audience’s search patterns. Keyword research tools such as Semrush and others can filter lists by modifiers to reveal the most popular.

Semrush’s Keyword Magic Tool reveals the most popular modifiers for “shoes.”

Adjust Semrush’s “Advanced filters” to see queries that contain more words.

“Advanced filters” reveal queries that contain more words.

Search Console

Regular expressions (regex) in Search Console can identify longer queries, such as fan-out searches from ChatGPT and other genAI platforms. In Search Console, go to “Performance,” click “Add filter,” choose “Query,” and “Custom (regex).”

Then type:

([^” “]*s){10,}?

This regex filters queries to those with more than 10 words. Change “10” to “5” or “25” to find queries longer than 5 or 25 words, respectively.

Regex in Search Console can identify longer queries, such as fan-out searches from ChatGPT and other genAI platforms.

Keyword Dos and Don’ts

Search engines no longer match queries to exact word strings on web pages, focusing instead on the searcher’s intent or meaning. Hence a query for “red shoes for women” could produce an organic listing for “maroon slippers for busy moms.”

Keyword optimization circa 2025 reflects this evolution.

• Avoid stuffing a page with keywords. Instead, enrich content with synonyms and related phrases.

• Don’t create a page with variations of a single keyword. Group pages by modifiers and optimize for the entire group.

• Include the main keyword in the page title and the H1 heading. Google could use either of those to create the all-important search snippet.

• Assign products to only one category. Don’t confuse Google (and genAI platforms) by creating multiple categories for the same item to target different keywords.

• Search Google (and genAI platforms) for your target query and study the results. Are there other opportunities, such as images and videos?

• Don’t force an exact match keyword if it’s awkward or grammatically incorrect. Ask yourself, “How would I search for this item?” In other words, write for people, not search engines.

So You Want To Paywall?

There are three inevitabilities in life. Death, taxes, and big tech companies dumping on the little guy. As zero-click searches reach an all-time high and content is stolen and repurposed for the gain of the almighty tech loser, there’s only one viable solution.

To paywall.

To create a value exchange that reduces reliance on third-party platforms. To become as self-sufficient as possible. Like an off-grid cabin or your mum’s basement, a paywall gives you a sense of security you just cannot put a price on.

As we’re all finding, any kind of reliance on these guys doesn’t put us in a good position. They do not want to send us traffic.

TL;DR

1. Subscriber revenue is intrinsically more valuable to a business because it is predictable. Subscription and advertiser revenue are not created equal.
2. Don’t paywall everything. Use dynamic/metered paywalls and leave high-reach, generally lower-quality platforms like Google Discover free for email signups.
3. Subscription success relies on your USP – whether that’s exclusive data, deep, niche insights, or a certain vibe – you have to stand out.
4. The customer experience and understanding of your audience matter. Create habit-forming connections and products. Become an essential part of their life.

But What About Our Traffic?

Your traffic will decline. But guess what? You’re already hemorrhaging clicks and have been for some time. And traffic doesn’t pay the bills.

The only way to sustain rankings over time is with high-quality engagement data. Navboost stores and uses 13 months of data to identify good vs bad clicks, click quality, the last longest click, and on-page interactions to establish the most relevant content. All at a query level.

Paywalls are not your friend when it comes to user engagement. Not for the masses. But for a small cohort of people who like you enough to pay, your engagement data will be excellent.

In an ultra-personalized world, you will still do well to the people who really matter.

We have data that pretty perfectly highlights the impact of a paywall on rankings. Over the course of three to four months in traditional search, your rankings start to steadily drop before settling in severe mediocrity. You’ve got to fight for every click. With great content, marketing, savviness. Everything.

In Google Discover – a highly personalized, click and engagement driven platform – this is even more pronounced. While Discover’s clickless traffic is lower quality, there will be a small cohort of highly engaged users that develop over time, you can target with a paywall.

Unpaywall for the masses, build your owned channels, and paywall for the highly engaged. The platform will take care of the personalization for you.

So, maximize your value exchange with ads and email signups for most users, but don’t neglect those with a high return rate.

There’s some psychology involved in all of this. When a brand becomes widely known for paywalling, I suspect the likelihood of a click goes down as users know what to expect. Or maybe what not to expect.

This likely perpetuates over time, so you should clarify what articles are free to air.

Is Our Content Good Enough?

To nail SEO bingo, it depends. It depends on what your value is in the market. There is a lot of free stuff out there already. But broadly rubbish. So as long as the bar keeps dropping, we’ll all be fine.

I am old-ish. I like words. Writing great content isn’t easy and is usurped in many cases by richer, more visually striking content. Content that satisfies all types of users. Scanners, deep readers, listeners and get the answer and go-ers.

In some ways, you can satisfy all types of users more effectively than ever. I think you have to hit three of the four Es of content creation. Make it resonate, be consistent, and understand your audience. Whatever you create stands a chance.

But that doesn’t mean creating great stuff is any easier. If you work for a traditional publisher, the chances are you’ve brought a spoon to a gun fight. The war for attention is being fought on all fronts, and straight words are losing.

Fortunately, not every subscription model relies on the quality of the prose. It might be that you have unique data, granular insights into a specific market, or are just a bloody good laugh.

Subscriptions come in all shapes and sizes.

Ultimately, it comes down to your market, marketing, positioning and your USP. You have to know and speak to your audience and you have to stand out. As Barry would say, if you’re forgettable, you’re doomed.

How Do We Know If People Will Pay?

When it comes to paying for news, some markets are far more “advance” than others. The Scandinavian market is light-years ahead of almost everyone else when it comes to paying for news. You have to do your research to understand:

• How many people currently pay for news?
• What demographic of person pays?
• How saturated is the market already?
• What is your niche?

While it doesn’t align perfectly, it’s not surprising that those most likely to pay for news have higher income levels. Higher disposable income tends to create an environment where people buy more stuff.

Shocking, I know.

But that doesn’t tell the whole story. Norwegian news outlets have (apparently) a long history of trust with their audience and have never had access to free multi-day newspapers. Ditto other Scandinavian countries. In an age of rubbish and spin, trust and E-E-A-T are more important than ever.

And while the UK sits in a pretty shocking-looking position, almost 24 million of us pay for a BBC license fee. That is, in essence, paying for news. Insert joke about BBC bias and woke cultural agendas here.

Cultural and societal factors really matter. As does your understanding of the market.

Important to note that according to Richard Reeves (AOP Director), Subscriptions have overtaken display advertising as the core source of digital revenue.

“Most heartening is what this represents as the wider information ecosystem fractures: audiences recognise the value of professional journalism and are willing to pay for it.”

In an era of slop, paying for something good is not a bad thing.

Macro And Micro Factors Are Influential

You can only control what you can control. But you shouldn’t dismiss the wider climate.

In the UK and arguably globally, there is a cost-of-living crisis. Globally, there have been a number of very significant geopolitical issues that affect the wider economy. Money doesn’t go as far as it once did, and most subscriptions are a luxury purchase.

Is a £20 or £30 monthly subscription more valuable than a £10 Netflix one? Or Spotify? These are questions you need to ask. Why would someone subscribe and stick around?

And we aren’t just competing with other publishers. While screen time and content consumption are at an all-time high, video consumption and the creator economy are booming.

It is quite literally a near half a trillion dollar market.

Not strengths for traditional publishers. While there have been some very good success stories in recent times (see Wired turning their journalists into individual subscription machines), legacy publishers need to adapt.

So your pricing strategy, customer service, and overall experience are hugely important. You are almost certainly going to be a nice-to-have. So make sure your customer journey and path to conversion are premium, and your audience feel listened to.

You need to speak to your audience. You don’t have to go into this blind. Forging real connections with people is not impossible and making them feel listened to will go a long way.

You can try to figure out what they really value, how much they’re willing to spend and what’s stopping them.

Should I Paywall Everything?

No. Content is designed to do different things, and not everything is a premium product. Whatever journalists will tell you. If you shut down your site entirely, you become too closed off an ecosystem in my opinion.

• Commercial Content: If you have affiliate-led content, paywalling is a questionable decision. It may not be wrong per se, but think about whether the pros outweigh the cons. Typically, it’s a good gateway drug for the rest of your content. And makes some money.
• Content You Can Get Elsewhere: Evergreen content of a comparable quality to what already exists in the wider corpus is not a profitable opportunity. I’d argue that leaving this free-to-air has more pros than cons. You can always unpaywall the 100 best albums of all time, but gate the richer, individual album reviews.
• Lower-Quality Platforms: A user that comes from a platform like Discover is far less likely to convert than someone who comes from organic search. So think about the role each platform plays in your content access ecosystem.
• Paywall Vs. Newsletter signup: It is far easier to convert people to a paying subscriber from a newsletter database than from an on-page paywall. And the user journey is far less interrupted. Building an owned channel is never a bad thing, so think about how engaged users are and whether an email would be a more effective starting point.

The Type Of Paywall Matters (Now More Than Ever)

LLMs do not respect paywalls. As it turns out, neither does Google.

I, for one, am stunned.

As of just a few months ago, the search giant asked that publishers with paywalls change the way they block content to help Google out. The lighter touch paywall solution (a JavaScript-based one) includes the full content in the server response.

“…Some JavaScript paywall solutions include the full content in the server response, then use JavaScript to hide it until subscription status is confirmed.

This isn’t a reliable way to limit access to the content. Make sure your paywall only provides the full content once the subscription status is confirmed.”

According to Google, they are struggling to determine the difference. So the problem is on us, not them. They (and I strongly suspect other LLMs) are ingesting this content and training their models on us whether we like it or not.

For those of you who haven’t heard of Common Crawl, it stores a corpus of open web data accessible to “researchers.” By researchers, we now mean tech bros who don’t want to pay for, surprisingly, anything.

According to their CEO;

“If you didn’t want your content on the internet, you shouldn’t have put your content on the internet.”

It doesn’t stop there either. Even if you block all non-whitelisted bots from accessing your site at a CDN level, you may have syndication partnerships in place. If so, it’s likely your content is making it out into the wider world.

The internet is not exactly a leakproof vessel. If you’re setting one up now, try to implement a server-side option.

What Is The Right Paywall For Me?

I have written about the types of paywall available to you and the pros and cons of each. Generally, I think a metered or dynamic paywall is the best option for most publishers. At the very least, a freemium model. Something that gives people enough to draw them in.

And you can’t exactly draw them in if you just hard paywall everything.

You have to think of this as a full-blown marketing strategy. You need to know where people come from. How much of your content they have consumed. Whether it’s better to show them a newsletter signup as opposed to a paywall.

It is absolutely worth knowing that over time, a strong email database will convert far more effectively than a hard paywall.

So encouraging free signups and taking a longer-term view to conversions (you’ll need a good customer journey here) may be far more effective.

How Can I Set One Up?

There are a number of paywall management options out there for publishers. Leaky Paywall, Zephr, Piano. There are plenty.

The best ones integrate with your existing tech stacks, have excellent personalization and customization options, deploy ad-blocking strategies, and run flexible gating strategies.

Larger publishers tend to go with enterprise-level options with deep analytics and CRM integrations. Smaller publishers can work with lighter touch, cheaper operators. You really just need to scope out what will work best for you.

Particularly when it comes to monthly costs and revenue share options.

How Can I Map The Impact?

You’ll need to establish a few key things:

• The average drop in traffic you expect to see.
• The subsequent loss of existing revenue (probably ad-related, but there may be some knock-on wider commercial impact).
• The average value of a subscription (and the expected conversion rate).
• Your customer LTV.

Focusing on Customer LTV shifts marketing from chasing traffic to profitable, loyal audience relationships. It makes businesses understand that not all audiences or subscriptions are created equal.

You generate more subs through paid media because the net is larger. But lots slip through the net. So you need a quality product (in both a product and marketing sense) alongside UX and customer service that reduces friction.

Search and owned channels are smaller, but far more likely to pay because they have taken an action to find you. In some cases, they actually want you in their inbox. The quality is higher, but the overall returns are lower.

So you just can’t treat everybody the same.

Closing Thoughts

Subscriber revenue is so valuable because it’s predictable. Subscription business models have boomed for that very reason. A pound of subscriber revenue is far more valuable than almost anything else, and it should be the focus of your business.

But that doesn’t mean you put all your eggs in one basket. You can have multiple subscription types on your website, and that can help you become habitual with all types of users. But you need to add value to their lives every day.

Puzzles, recipes, short and long-form videos, et al.

Businesses make money in many ways. A diverse business is resilient. Resilient to macro and micro factors that will decimate some publishers over the next few years. So talk to your audience, trial new ways of adding value, and commit when one works. Become habitual.

And, shock horror, people want to belong to something. So while the digital experience is crucial, making an effort to connect with people IRL matters.

More Resources:

This post was originally published on Leadership in SEO.

Featured Image: beast01/Shutterstock