A Guide To Enterprise-Level Migrations (100k + URLs) via @sejournal, @TaylorDanRW

An enterprise website migration is no small feat.

We’re talking hundreds of thousands of URLs and years of SEO equity on the line.

To pull it off without traffic loss, you need a solid redirect strategy.

With the right approach, you can migrate an enterprise website without losing traffic or search rankings.

If you stay organized, leverage tools to scale, and pay attention to details, you will have done everything you can to ensure business continuity in the short, medium, and long term pertaining to organic performance.

Aside from the technical aspects of migration, an enterprise migration, more often than not, comes with the added pressures of:

• Strong levels of C-level/VP-level attention and communications.
• Multiple project teams and stakeholders making SEO-impacting decisions.
• SEO pros needing to be involved in “non-traditional” SEO calls and planning meetings.

In a large site migration, there is also the increased potential for something known as “migration lag.”

What Is Migration Lag?

Migration lag refers to the time period after launching a new website where traffic and rankings drop as search engines discover and index the new site.

For huge enterprise sites with hundreds of thousands of URLs, this lag can last for months.

To minimize migration lag, you must have a solid redirect strategy before the new site launches. This means:

• Prioritizing redirects for high-traffic and high-value pages. Focus on redirecting pages that drive the most traffic and revenue first.
• Using wildcards to redirect categories of pages. For example, redirect /product/* to /new-site/all-products/.
• Including URL parameters in redirects. Make sure redirects pass on any query parameters, like /product/123?color=red to /new-site/product/123?color=red.
• Breaking redirect chains. If a page has been redirected multiple times, point the final redirect to the new destination URL.
• Redirecting backlinks. Find all links pointing to the old site and set up redirects so they point to the proper new pages. This preserves the link equity you’ve built up.
• Accounting for recent redirects. If you’ve done any redirects in the past six months, set up new redirects to point those pages to the proper new URLs.

With technical SEO savvy and patience, you can navigate an enterprise website migration with minimal traffic and rankings loss.

Stay on top of your redirects and keep optimizing and reacting to your data and Google’s ever-changing search engine results pages (SERPs), and search traffic will return to normal.

Soft-Launch Pre-Migration

In June 2023, John Mueller floated the idea of launching a new domain “early” before the official migration switchover.

This was interesting, as the general best practice narrative has been to not let the new domain be open to crawling before the migration switchover date.

As with any new recommendation, this is something I’ve tested since on personal project sites and with client moves.

Testing has shown that indexing has happened faster for the new domain, especially when compared to the domains in my “How Long Should A Migration Take” study.

In the Google Search Console screenshot below, I migrated a domain on January 28, but I put the new domain live and crawlable/indexable from January 21.

By February 1, the new domain was 100% indexed, and Google had even crawled and processed all the /feed URLs that were set to nodindex.

Screenshot from Google Search Console, February 2025

While this was a small website (1,300 URLs), the data was similar to other domain migrations and subdomain to subfolder migrations I’ve taken this approach with.

The most common pushback I’ve had to this approach has been the wider business desire to “make a splash” with PR around the launch and the chance of an existing customer finding the new site early. If they share the new site, the potential problems this could cause can diminish any benefits gained.

The second most common pushback, which is valid, is if there have been substantial changes to content, product, or brand that need to remain under embargo until the scheduled launch date.

Defining The Migration Strategy

Once you’ve audited your existing site and redirects, it’s time to map out how you want to handle the migration.

The strategy you develop now will determine how seamless this transition is for both your users and search engines.

Define Goals

What do you want to achieve with this migration? Are you aiming to consolidate domains, move to a new content management system (CMS), restructure content, or combine?

Be very clear on your objectives so you can develop the best approach.

Prioritize Redirects

With hundreds of thousands of URLs, you’ll need to determine which redirects are most critical to implement first. Focus initially on:

• Your most important pages (homepage, product pages, etc.).
• Pages that generate a substantial amount of website leads/revenue, either directly or indirectly.
• Pages that generate the most organic traffic for the website.
• Pages with strong backlink profiles. Those that are crawled frequently by Google/other search engines should be prioritized above those with bigger backlink clusters – but this is an objective measure you will need to determine.

Once the high-priority redirects are handled, work your way down from there. Don’t worry about redirecting every single URL right away.

As long as you have the majority of important pages and traffic accounted for, the remaining redirects can be added over time.

A great way to prioritize redirects, is to create a dashboard of all relevant data you wish to consider and prioritize by (such as the examples I’ve given above) and creating a matrix with RANK.EQ in Google Sheets, and then a prioritization categorizer.

The example below is a very simplified version of this. First, you want to collate all your data at the URL level:

Image by author, February 2025

You then want to rank these values against their individual metric data sets. To do this, you use =RANK.EQ(VALUE,VALUE RANGE).

This lets you see which URLs are in the higher percentile and which ones are in the lower percentile:

Image by author, February 2025

You then want to automate batch assignment, and this requires three steps.

First, a “reverse RANK.EQ”, which would be:

=COUNT(A:A) - RANK.EQ(A1, A:A) + 1

Which will tell you which URLs are “the best” based on all four metric ranks combined:

Image by author, February 2025

From here, you can either convert the Overall EQ to percentages in another column, and then run a rule against them that if they are =< or => certain thresholds, they fall into different batches.

You can also split the rows up by volume ordered by the Overall EQ if you have redirect limits (like when moving to Salesforce Commerce Cloud or SAP Hybris; read more below).

Map Content And URL Structure

Determine how you want to reorganize or restructure your content on the new site.

Map out which existing URLs will redirect to which new destinations. Group related content and consolidate where possible.

The new information architecture should be intuitive and user-friendly.

Redirect Types

For the bulk of redirects, use 301 permanent redirects.

In some cases, temporary 302 redirects may make sense, especially if the page content is still being migrated.

Be very careful when using wildcards, and always do spot checks to ensure there are no 404 errors. Redirect parameters whenever possible to avoid duplicate content issues.

Backlinks

Make a list of any pages with strong backlink profiles and ensure they are redirected properly. Reach out to webmasters linking to those pages and let them know the new URL to see if they will update the link on their page.

This helps to preserve the SEO value built up over time.

With careful planning and strategic prioritizing, you can migrate an enterprise website and put the necessary redirects in place without (too much) chaos. But go slowly; this is not a task to rush!

Think through each step and check your work along the way.

Establishing The Migration Project Timelines

When managing a large website migration, establishing realistic timelines is crucial.

Trying to redirect hundreds of thousands of URLs in a short timeframe is a recipe for disaster.

You need to plan ahead and be strategic in how you phase the work.

It’s also very important that migration timelines are a collaborative effort involving all stakeholders.

Far too often, the business determines an arbitrary deadline without taking into account the feasibility of all teams to complete all necessary actions comfortably in time.

Avoid Phased/Partial Migrations

Avoiding phased or partial migrations is crucial when managing redirects for an enterprise website. Piecemealing your migration will only create more work and headaches down the road.

I worked on a migration in the past two years that was consolidating multiple domains (products) under a new umbrella domain, and the original plan was to do one after the other in a phased approach.

More than a year later, the second domino still hasn’t fallen. Google has started to rank the umbrella domain for products in the group it isn’t optimized for – causing internal domain cannibalization and performance issues as the brand entity is “fractured” across multiple domains.

Prior to this, I’d never witnessed a phased or partial migration mitigate the risks to the performance that the cautious decision-makers felt it would.

Do It All At Once

The best approach is to redirect all URLs at the same time. This ensures:

• No pages are left orphaned without a redirect in place.
• There are no redirect chains created that need to be cleaned up later. Redirect chains can negatively impact SEO and user experience.
• All backlinks point to the proper new destination page. If done in phases, old pages may accumulate new backlinks that then need to be redirected.

Setting Up 301 Redirects At Scale

At an enterprise level, setting up 301 redirects for tens or hundreds of thousands of URLs requires some strategic planning.

Here are some tips for tackling this at scale:

Using Wildcards And Handling Parameter URLs

When managing redirects for an enterprise website, wildcards and parameters become your best friends. With so many URLs, creating individual redirects for each would be an endless task.

Wildcards allow you to redirect groups of pages at once.

Say you have product pages like /product/abc123, /product/def456, /product/ghi789. You can set up a wildcard redirect like /product/* to point to the new /products page.

This single redirect will capture all product pages and send visitors to the right place.

Parameters, like IDs, SKUs, or dates, often change when site content gets updated or reorganized.

Rather than tracking down each instance of an old parameter to redirect it, use a redirect that includes the parameter.

For example, if you have a URL like /blog/post?id=123 that is now /news/story/123, set up the redirect /blog/post?id= to point to /news/story/.

This will catch any page with that parameter pattern and send visitors to the new structure.

When used properly at an enterprise scale, wildcards and parameters can:

• Save countless hours of manual redirect creation and maintenance.
• Ensure no page is left behind during a migration or site architecture change.
• Continue to capture new pages that match the pattern as the site grows and evolves.

Be very careful when using wildcards and parameters in your redirects. Test them thoroughly to ensure no unintended pages are caught in the net.

Monitor them regularly, even after launch, to catch any issues early. Used responsibly, though, they are indispensable tools for managing redirects at an enterprise level.

Breaking Redirect Chains

Redirect chains can easily form when you have a high volume of redirects on an enterprise website.

A redirect chain occurs when a URL redirects to another URL that also redirects, creating a chain of multiple redirects to reach the final destination page.

To avoid redirect chains, you’ll need to trace back through your recent redirect history to find the original source URL. Once you identify the initial URL that started the chain, redirect it directly to the final destination page.

This will cut out all the middle redirects in the chain and provide a much better user experience.

• Check your server log files to view URL redirect histories from the past three to six months. Look for any patterns of the same URL redirecting multiple times.
• Use a redirect crawler tool to automatically detect redirect chains on your site. These tools will crawl your site and log any series of multiple redirects for the same URL.
• For recent redirects less than 180 days old, double-check that the original URL is now redirecting properly to the correct final destination. Newer redirects have a higher chance of issues, so verifying them will help avoid future problems.
• If you discover broken redirect chains, fix them by redirecting the initial source URL directly to the last destination URL in the chain. Remove any middle redirects that are no longer needed.
• Test all fixes to ensure the redirect chain is fully broken and the user experience is improved. Check that SEO rankings and traffic have stabilized for the URLs involved.

By diligently detecting and breaking redirect chains, you’ll provide a much better overall experience for your users and site visitors.

Your enterprise website will function more efficiently, and you’ll avoid potential drops in search rankings and traffic.

Historic Redirects

When migrating an enterprise website, it’s easy to forget about redirects that were already in place. These historic redirects, especially those under six months old, still need to be accounted for to avoid traffic loss.

As you audit your site’s current redirects, make a list of any that point to pages that will be changing or removed in the migration.

These redirects will need to be updated to point to the new destination URLs. Some things to look for include:

• Temporary event pages that now redirect to a general section.
• Product pages that now redirect to an updated model.
• Blog posts that redirect to a category archive.

Double-check that any historic redirects over six months old still need to be in place. Some may be sending signals to search engines that are no longer needed.

Removing unnecessary historic redirects will also help to simplify your site’s redirect structure and make it easier to manage going forward.

When setting up your migration’s redirect plan, be sure to factor in updating any historic redirects to their new destination URLs.

Leaving these behind could result in lost traffic and rankings for important pages on your site.

Staying on top of your enterprise website’s historic and new redirects during migration is key to a successful transition with minimal SEO impact.

Overcoming Redirect Limits

If you have an enterprise website with hundreds of thousands of pages, you may run into issues with redirect limits from your CMS or ecommerce platform.

Many systems like SAP Hybris and Salesforce Commerce Cloud cap the number of redirects you can have at 50,000 to 100,000. For a major website migration (especially enterprise ecommerce websites), this likely won’t cut it.

To get around these constraints, you’ll need to get creative. A few options to consider:

• Use wildcard redirects to capture categories of pages. For example, redirect /products/* to /shop/*. This single redirect will capture all pages that start with /products.
• Exclude parameters from redirects when possible. If you have pages like /product-name?color=red and /product-name?size=large, redirect only /product-name to the new URL. The parameters are often not indexed or linked to, so you can leave them out of the redirect.
• Break up redirect chains. If you have a series of three+ redirects for a single page, break up the chain and create direct redirects from the initial URLs to the final destination. Historically, chained redirects were thought to pass along link juice, but this has been proven false. Keep redirects as direct as possible.
• Prioritize mission-critical pages. When you start to reach the redirect limit, focus on redirecting pages that drive significant traffic and revenue. You can leave less important pages unredirected or with a 404 error temporarily.
• Ask your CMS vendor about increasing limits. Many systems will increase redirect limits on an enterprise website if you ask and explain your needs. Be prepared to pay additional fees for this add-on.

With creative thinking and persistence, you can overcome most redirect limits and complete an enterprise website migration without losing a big chunk of your organic traffic.

The key is having a well-thought-out redirect strategy and implementing it well before you hit your CMS’s limits.

Benchmarking Organic Performance (Traffic, Rankings, Indexation)

Once the redirects have been implemented, it’s time to see how your organic traffic and rankings have been impacted.

Benchmarking Your Progress

This will help determine if any further optimization is needed. Here are a few key metrics to monitor:

• Organic search traffic. Compare traffic from major search engines like Google before and after the migration. Expect some initial drop in traffic, but it should start recovering within one to two months. If traffic is still significantly lower after three months, revisit your redirect implementation.
• Keyword rankings. Track the rankings of your most important keywords to see if their positions have changed. Drops in ranking are common after a migration, but rankings should start improving again over time as search engines recrawl your new site. Major drops that don’t improve could signal redirect or content issues that need to be addressed.
• Indexation. Use a tool like Google Search Console to check how much of your new site has been indexed. A large, complex site can take three to six months for Google to fully recrawl and re-index. Look for steady increases in indexation over time. If indexation stalls or drops, there may be technical issues preventing Google from accessing parts of your site.
• 404 errors. Monitor your 404 errors to ensure any broken links are redirecting properly. High numbers of 404s, especially old URLs, indicate redirects that need to be created or fixed.
• Backlinks. Do a backlink audit to verify that any links pointing to your old site now redirect to the proper new URLs. Failure to redirect backlinks is a common cause of traffic and ranking loss after a website migration.

Regularly benchmarking these key metrics will give you valuable insight into how well your enterprise website migration and redirect implementation is going.

Make adjustments as needed to get your new site’s organic performance back on track.

Communicating Migration Performance To The C-Level

Communicating migration performance to leadership is crucial for continued support and investment in your enterprise website.

Even if the migration itself goes smoothly, problems can arise after launch if the C-suite isn’t on board.

Set Clear Expectations

Before the migration, sit down with executives and set concrete goals and key performance indicators (KPIs) for the new site.

Not all metrics need to matter directly to SEO, but giving the C-level more data and clarity can help prevent knee-jerk reactions and bad decisions from being imposed on the migrations team.

Be transparent that there may be an initial dip in metrics as the new site establishes itself. Having targets will help determine if the migration is meeting business needs after things settle in.

Share Detailed Reports

In the months following the migration, provide regular reports on how the new site performs compared to the old site and the established KPIs.

Compare these same metrics from the old site to give context on progress. Be open about any issues, and have solutions and next steps ready to propose.

It often helps to create a Looker Studio report so the C-level has instant access to data and feels as though they have some control over the situation.

Finally, Don’t Forget To Highlight Wins

While reporting on challenges is important, it is also important to showcase successes from the migration.

Promoting wins, big and small, demonstrates the value of the investment in the migration and builds confidence in your team.

Keeping leadership regularly informed about how the new enterprise website is performing is essential.

With open communication and a mix of progress reports and wins, executives will remain supportive and engaged in optimizing the site to achieve the best results.

More Resources:

Featured Image: Munthita/Shutterstock

How To Use XML Sitemaps To Boost SEO via @sejournal, @jes_scholz

What was considered best practice yesterday does not hold true today and this is especially relevant when it comes to XML sitemaps, which are almost as old as SEO itself.

The problem is, it’s time-consuming to sort valuable advice from all the misinformation on forums and social media about how to optimize XML sitemaps

So, while most of us recognize the importance of submitting sitemaps to Google Search Console and Bing Webmaster Tools, as well as in the robots.txt file – for faster content discovery and refresh, more efficient crawling of SEO-relevant pages, and valuable indexing reporting to identify SEO issues – the finer details of implementing sitemaps to improve SEO performance may be missed.

Let’s clear up the confusion and dive into the current best practices for sitemap optimization.

In this article, we cover:

• What is an XML sitemap?
• How to create a sitemap.
• Valid XML sitemap format.
• Types of sitemaps.
• Optimization of XML sitemaps.
• XML sitemap best practice checklist.

What Is An XML Sitemap?

An XML sitemap is a file that lists all of your website’s URLs.

It acts as a roadmap to tell the crawlers of indexing platforms (like search engines, but also large language models (LLMs)) what content is available and how to reach it.

Image from author, February 2025

In the example above, a search engine will find all nine pages in a sitemap with one visit to the XML sitemap file.

On the website, it will have to jump through five internal links on five pages to find page 9.

This ability of XML sitemaps to assist crawlers in faster indexing is especially important for websites that:

• Have thousands of pages and/or a deep website architecture.
• Frequently add new pages.
• Frequently change the content of existing pages.
• Suffer from weak internal linking and orphan pages.
• Lack of a strong external link profile.

Even though indexing platforms could technically find your URLs without it, by including pages in an XML sitemap, you’re indicating that you consider them to be quality landing pages.

And while there is no guarantee that an XML sitemap will get your pages crawled faster, let alone indexed or ranked, submitting one certainly increases your chances.

How To Create A Sitemap

There are two ways to create a sitemap: Static sitemaps that must be manually updated, or dynamic sitemaps that are updated in real-time or by a regular cron job.

Static sitemaps are simple to create using a tool such as Screaming Frog.

The problem is that as soon as you create or remove a page, your sitemap is outdated. If you modify the content of a page, the sitemap won’t automatically update the lastmod tag.

So, unless you love manually creating and uploading sitemaps for every single change, it’s best to avoid static sitemaps.

Dynamic XML sitemaps, on the other hand, are automatically updated by your server to reflect relevant website changes.

To create a dynamic XML sitemap you can do one of the following:

• Ask your developer to code a custom script, being sure to provide clear specifications.
• Use a dynamic sitemap generator tool.
• Install a plugin for your content management system (CMS), for example, Yoast plugin for WordPress.

Valid XML Sitemap Format

Image from author, February 2025

Your sitemap needs three items to introduce itself to indexing platforms:

• XML Version Declaration: Specifies the file type to inform indexing platforms what they can expect from the file.
• UTF-8 Encoding: Ensures all the characters used can be understood.
• Specify The Namespace: Communicates what rules the sitemap follows. Most sitemaps use the “http://www.sitemaps.org/schemas/sitemap/0.9” namespace to show that the file conforms to standards set by sitemaps.org.

This is followed by a URL container for each page. In a standard XML sitemap, there are only two tags that should be included for a URL:

1. Loc (a.k.a. Location) Tag: This compulsory tag contains the absolute, canonical version of the URL location. It should accurately reflect your site protocol (http or https) and if you have chosen to include or exclude www.
2. Lastmod (a.k.a. Last Modified) Tag: An optional but highly recommended tag to communicate the date and time the page was published or the last meaningful change. This helps indexing platforms understand which pages have fresh content and prioritize them for crawling.

Google’s documentation on sitemaps states:

“Google uses the value if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate. The value should reflect the date and time of the last significant update to the page. For example, an update to the main content, the structured data, or links on the page is generally considered significant, however an update to the copyright date is not.”

Bing’s documentation agrees on the importance of the lastmod tag:

“The “lastmod” tag is used to indicate the last time the web pages linked by the sitemaps were modified. This information is used by search engines to determine how frequently to crawl your site, and to decide which pages to index and which to leave out.”

Mistakes, such as updating the value when the sitemap is generated rather than when the individual page was last modified, or worse, trying to manipulate crawlers by updating the date without significantly altering the page, may result in this signal being ignored by search engines for your website. Damaging your ability to have your content efficiently crawled.

Do not include the Changefreq (a.k.a. Change Frequency) Tag or priority tag. Once upon a time, these hinted at how often to crawl, but are now ignored by search engines.

Types Of Sitemaps

There are many different types of sitemaps. Let’s look at the ones you actually need.

XML Sitemap Index

XML sitemaps have a couple of limitations:

• A maximum of 50,000 URLs.
• An uncompressed file size limit of 50 MB.

Sitemaps can be compressed using gzip to save bandwidth for your server. But once unzipped, the sitemap still can’t exceed either limit.

Whenever you exceed either limit, you will need to split your URLs across multiple XML sitemaps.

Those sitemaps can then be combined into a single XML sitemap index file, often named sitemap-index.xml. Essentially, it is a sitemap for sitemaps.

You can create multiple sitemap index files. But be aware that you cannot nest sitemap index files.

For indexing platforms to easily find every one of your sitemap files, you will want to:

• Submit your sitemap index to Google Search Console and Bing Webmaster Tools.
• Specify your sitemap or sitemap index URL(s) in your robots.txt file. Pointing indexing platforms directly to your sitemap as you welcome them to crawl.

Image from author, February 2025

Image Sitemap

Image sitemaps were designed to improve the indexing of image content, originally offering additional tags.

In modern-day SEO, however, it’s best practice to utilize JSON-LD schema.org/ImageObject markup to call out image properties to indexing platforms, as it provides more attributes than an image XML sitemap.

Because of this, a dedicated XML image sitemap is unnecessary. Simply add the image XML namespace declaration and the image tags directly to the main XML sitemap within the associated URL.

Image from author, February 2025

Know that images don’t have to be on the same domain as your website to be submitted in a sitemap. You can use a CDN as long as it’s verified in Google Search Console.

Video Sitemap

Similar to images, Google says video sitemap tags can be added within an existing sitemap.

However, unlike images, video extensions in sitemaps offer a multitude of additional tags.

Image from author, February 2025

If you leverage these tags extensively, consider a dedicated video sitemap.

Adding these extensions increases the file size of your sitemap significantly and may lead to you exceeding the file size limits.

Either method will help Google discover, crawl, and index your video content as long as the video is related to the content of the host page and is accessible to Googlebot.

While Bing does support video extensions in XML sitemaps, Fabrice Canel confirmed to me that they prefer submission via IndexNow. Although Bing’s documentation still mentions the mRSS format.

Google News Sitemap

Google News sitemaps can only be used for article content that was created in the last two days. Once the articles are older than 48 hours, remove the URLs from the sitemap.

Again, while Google News sitemap tags can be included in your regular sitemap, this is not recommended.

Unlike for image and video, only Google leverages the news sitemap extension, not Bing or other indexers.

Image from author, February 2025

Contrary to some online advice, Google News sitemaps don’t support image URLs.

HTML Sitemap

XML sitemaps take care of indexing platform needs. HTML sitemaps were designed to assist human users in finding content.

The question becomes: If you have a good user experience and well-crafted internal links, do you need an HTML sitemap?

Check the page views of your HTML sitemap in Google Analytics. Chances are, it’s very low. If not, it’s a good indication that you need to improve your website navigation.

HTML sitemaps are generally linked in website footers. Taking link equity from every single page of your website.

Ask yourself. Is that the best use of that link equity? Or are you including an HTML sitemap as a nod to legacy website best practices?

If few humans use it, and indexing platforms don’t need it as you have strong internal linking and an XML sitemap, does that HTML sitemap have a reason to exist? I would argue no.

XML Sitemap Optimization

XML sitemap optimization involves how you structure your sitemaps and what URLs are included.

How you choose to do this impacts how efficiently indexing platforms crawl your website and, thus, your content visibility.

Here are four ways to optimize XML sitemaps:

1. Only Include SEO Relevant Pages In XML Sitemaps

An XML sitemap is a list of pages you want to be crawled (and subsequently given visibility to by indexing platforms), which isn’t necessarily every page of your website.

A bot arrives at your website with an “allowance” for how many pages it will crawl.

The XML sitemap indicates that you consider the included URLs more important than those that aren’t blocked but not in the sitemap.

You’re using it to tell indexing platforms, “I’d really appreciate it if you’d focus on these URLs in particular.”

To help them crawl your site more intelligently and reap the benefits of faster (re)indexing, do not include:

• 301 redirect URLs.
• 404 or 410 URLs.
• Non-canonical URLs.
• Pages with noindex tags.
• Pages blocked by robots.txt.
• Paginated pages.
• Parameter URLs that aren’t SEO-relevant.
• Resource pages accessible by a lead gen form (e.g., white paper PDFs).
• Utility pages that are useful to users, but not intended to be landing pages (login page, contact us, privacy policy, account pages, etc.).

I’ve seen recommendations to add 3xx, 4xx, or non-indexable pages to sitemaps in the hope it will speed up deindexing.

But similar to manipulation of the last mod date, such attempts to get these pages processed faster may result in the sitemaps being ignored by search engines as a signal, damaging your ability to have your valuable content efficiently crawled.

But remember, Google is going to use your XML submission only as a hint about what’s important on your site.

Just because it’s not in your XML sitemap doesn’t necessarily mean that Google won’t index those pages.

2. Ensure Your XML Sitemap Is Valid

XML sitemap validators can tell you if the XML code is valid. But this alone is not enough.

There might be another reason why Google or Bing can’t fetch your sitemap, such as robots directives. Third-party tools won’t be able to identify this.

As such, the most efficient way to ensure your sitemap is valid is to submit it directly to Google Search Console and Bing Webmaster Tools.

Image from author, February 2025

When valid in GSC and BWT, you will see the green “Success” status.

Image from author, February 2025

If you get a red message instead, click on the error to find out why, fix it, and resubmit.

But in Google Search Console and Bing Webmaster Tools, you can do so much more than simple validation.

3. Leverage Sitemap Reporting For Indexing Analysis

Image from author, February 2025

Say you submit 80,000 pages all in one sitemap index, and 9,000 are excluded by both Google and Bing.

Sitemap reporting will help you to understand overarching why, but provides limited reporting on which URLs are problematic.

So, while it’s valuable information, it’s not easily actionable. You need to discover which types of pages were left out.

What if you use descriptive sitemap names that reflect the sections of your website – one for categories, products, articles, etc.?

Image from author, February 2025

Then, we can drill down to see that 7,000 of the 9,000 non-indexed URLs are category pages – and clearly know where to focus attention.

This can also be done within a sitemap index file.

Now, I know both Google and Schema.org show examples encouraging numbered naming. So, you may have ended up in a /sitemap-products-index.xml file with something like this:

• /products-1.xml
• /products-2.xml

Which is not the most insightful naming convention. What if we break it down into parent categories? For example:

• /products-mens.xml
• /products-womens.xml
• /products-kids.xml

And if your website is multilingual, be sure to leverage language as an additional separation layer.

Such smart structuring of sitemaps to group by page type allows you to dive into the data more efficiently and isolate indexing issues.

Just remember, for this to effectively work, sitemaps need to be mutually exclusive, with each URL existing in only one sitemap. The exception is the Google News sitemap.

4. Strategize Sitemap Size

As mentioned before, search engines impose a limit of 50,000 URLs per sitemap file.

Some SEO specialists intentionally reduce this number, say to 10,000. This can be helpful to speed up indexing.

However, you can only download 1,000 URLs in GSC. So, if 2,000 URLs in a certain sitemap are not indexed, you can only access half of them. If you are trying to do content cleanup, this will not be enough.

Image from author, February 2025

To gain full visibility on all URLs causing issues, break sitemaps down into groups of 1,000.

The obvious downside is that this has a higher setup time as all URLs need to be submitted in Google Search Console and Bing Webmaster Tools. This may also require high levels of ongoing management.

XML Sitemap Best Practice Checklist

Do invest time to:

✓ Dynamically generate XML sitemaps.

✓ Compress sitemap files.

✓ Use a sitemap index file.

✓ Include the & tags.

✓ Use image tags in existing sitemaps.

✓ Use video and Google News sitemaps if relevant.

✓ Reference sitemap URLs in robots.txt.

✓ Submit sitemaps to both Google Search Console and Bing Webmaster Tools.

✓ Include only SEO-relevant pages in XML sitemaps.

✓ Ensure URLs are included only in a single sitemap.

✓ Ensure the sitemap code is error-free.

✓ Group URLs in descriptively named sitemaps based on page type.

✓ Strategize how to break down large sitemap files.

✓ Use Google Search Console and Bing Webmaster Tools to analyze indexing rates.

Now, go check your own sitemaps and make sure you’re doing it right.

More Resources:

Featured Image: BEST-BACKGROUNDS/Shutterstock

Google’s JavaScript SERPs Impact Trackers, AI

Google’s search engine results pages now require JavaScript, effectively “hiding” the listings from organic rank trackers, artificial intelligence models, and other optimization tools.

The world’s most popular search engine began requiring JavaScript on search pages last month. Google stated the move aimed to protect its services from bots and “abuse,” perhaps a thinly veiled allusion to competitive AI.

These changes could complicate search engine optimization in at least three ways: rank tracking, keyword research, and AI visibility.

Google Search now requires browsers to have JavaScript enabled.

Impact of JavaScript

Web crawlers can scrape and index JavaScript-enabled pages even when the JavaScript itself renders the content. Googlebot does this, for example.

A web-scraping bot grabs the content of an HTML page in four steps, more or less:

• Request. The crawler sends a simple HTTP GET request to the URL.
• Response. The server returns the HTML content.
• Parse. The crawler parses (analyzes) the HTML, gathering the content.
• Use. The content is passed on for storage or use.

For example, before the JavaScript switch, bots from Ahrefs and Semrush crawled Google SERPs. A bot could visit the SERP for, say, “men’s running shoes,” parse the HTML, and use the data to produce rank-tracking and traffic reports.

The process is relatively more complicated with JavaScript.

• Request. The crawler sends a simple HTTP GET request to the URL.

• Response. The server returns a basic HTML skeleton, often without much content (e.g.,

  ).

• Execute. To run the JavaScript and load dynamic content., the crawler renders the page in a headless browser such as Puppeteer, Playwright, or Selenium.

• Wait. The crawler waits for the page to load, including API calls and data updates. A few milliseconds might seem insignificant, but it slows down the crawlers and adds costs.

• Parse. The crawler parses the dynamic and static HTML, gathering the content as before.

• Use. The content is passed on for storage or use.

The two additional steps — Execute and Wait — are far from trivial since they require full browser simulation and thus much more CPU and RAM. Some have estimated that JavaScript-enabled crawling takes three to 10 times more computing resources than scraping static HTML.

It is worth clarifying that Google does not render the entire SERP with JavaScript, instead requiring that visitors’ browsers enable JavaScript — essentially the same impact.

The time and resources to crawl a SERP vary greatly. Hence one cannot easily assess the impact of Google’s new JavaScript requirement on one tool or another other than an educated guess.

Rank tracking

Marketers use organic rank-tracking tools to monitor where a web page appears on Google SERPs — listings, featured snippets, knowledge panels, local packs — for target keywords.

Semrush, Ahrefs, and other tools crawl millions, if not billions, of SERPs monthly. Rendering and parsing those dynamic results pages could raise costs significantly, perhaps fivefold.

For marketers, this potential increase might mean tracking tools become more expensive or relatively less accurate if they crawl SERPs infrequently.

Keyword research

Google’s JavaScript requirement may also impact keyword research since identifying relevant, high-traffic keywords could become imprecise and more costly.

These changes may force marketers to find other ways to identify content topics and keyword gaps. Kevin Indig, a respected search engine optimizer, suggested that marketers turn to page- or domain-level traffic metrics if keyword data becomes unreliable.

AI models

The hype surrounding AI engines reminds me of voice search a few years ago, although the former is becoming much more transformative.

Likely AI models crawled Google results to discover pages and content. An AI model asked to find the best running shoe for a 185-pound male might scrape a Google SERP and follow links to the top 10 sites. Thus some marketers expected a halo effect from ranking well on Google.

But AI models must now spend extra time and computing power to parse Google’s JavaScript-driven results pages.

Wait and Adapt

As is often the case with Google’s changes, marketers must wait to gauge the JavaScript effect, but one thing is certain: SEO is changing.