What Can Log File Data Tell Me That Tools Can’t? – Ask An SEO via @sejournal, @HelenPollitt1

For today’s Ask An SEO, we answer the question:

“As an SEO, should I be using log file data, and what can it tell me that tools can’t?”

What Are Log Files

Essentially, log files are the raw record of an interaction with a website. They are reported by the website’s server and typically include information about users and bots, the pages they interact with, and when.

Typically, log files will contain certain information, such as the IP address of the person or bot that interacted with the website, the user agent (i.e., Googlebot, or a browser if it is a human), the time of the interaction, the URL, and the server response code the URL provided.

Example log:

6.249.65.1 - - [19/Feb/2026:14:32:10 +0000] "GET /category/shoes/running-shoes/ HTTP/1.1" 200 15432 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36" 

• 6.249.65.1 – This is the IP address of the user agent that hit the website.
• 19/Feb/2026:14:32:10 +0000 – This is the timestamp of the hit.
• GET /category/shoes/running-shoes/ HTTP/1.1 – The HTTP method, the requested URL, and the protocol version.
• 200 – The HTTP status code.
• 15432 – The response size in bytes.
• Mozilla/5.0 (Macintosh; Intel Mac OS X 14_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 – The user agent (i.e., the bot or browser that requested the file)

What Log Files Can Be Used For

Log files are the most accurate recording of how a user or a bot has navigated around your website. They are often considered the most authoritative record of interactions with your website, though CDN caching and infrastructure configuration can affect completeness.

What Search Engines Crawl

One of the most important uses of log files for SEO is to understand what pages on our site search engine bots are crawling.

Log files allow us to see which pages are getting crawled and at what frequency. They can help us validate if important pages are being crawled and whether often-changing pages are being crawled with an increased frequency compared to static pages.

Log files can be used to see if there is crawl waste, i.e., pages that you don’t want to have crawled, or with any real frequency, are taking up crawling time when a bot visits a site. For example, by looking at log files, you may identify that parameterized URLs or paginated pages are getting too much crawl attention compared to your core pages.

This information can be critical in identifying issues with page discovery and crawling.

True Crawl Budget Allocation

Log file analysis can give a true picture of crawl budget. It can help with the identification of which sections of a site are getting the most attention, and which are being neglected by the bots.

This can be critical in seeing if there are poorly linked pages on a site, or if they are being given less crawl priority than those sections of the site with less importance.

Log files can also be helpful after the completion of highly technical SEO work. For example, when a website has been migrated, viewing the log files can aid in identifying how quickly the changes to the site are being discovered.

Through log files, it’s also possible to determine if changes to a website’s structure have actually aided in crawl optimization.

When carrying out SEO experiments, it is necessary to know if a page that is a part of the experiment has been crawled by the bots or not, as this can determine whether the test experience has been seen by them. Log files can give that insight.

Crawl Behavior During Technical Issues

Log files can also be useful in detecting technical issues on a website. For example, there are instances where the status code reported by a crawling tool will not necessarily be the status code that a bot will receive when hitting a page. In that instance, log files would be the only way of identifying that with certainty.

Log files will enable you to see if bots are encountering temporary outages on the site, but also how long it takes them to re-encounter those same pages with the correct status once the issue has been fixed.

Bot Verification

One very helpful feature of log file analysis is in distinguishing between real bots and spoofed bots. This is how you can identify if bots are accessing your site under the guise of being from Google or Microsoft, but are actually from another company. This is important because bots may be getting around your site’s security measures by claiming to be a Googlebot, whereas, in fact, they are looking to carry out nefarious actions on your site, like scraping data.

By using log files, it’s possible to identify the IP range that a bot came from and check it against the known IP ranges of legitimate bots, like Googlebot. This can aid IT teams in providing security for a website without inadvertently blocking genuine search bots that need access to the website for SEO to be effective.

Orphan Pages Discovery

Log files can be used to identify internal pages that tools didn’t detect. For example, Googlebot may know of a page through an external link to it, whereas a crawling tool would only be able to discover it through internal linking or through sitemaps.

Looking through log files can be useful for diagnosing orphan pages on your site that you were simply not aware of. This is also very helpful in identifying legacy URLs that should no longer be accessible via the site but may still be crawled. For example, HTTP URLs or subdomains that have not been migrated properly.

What Other Tools Can’t Tell Us That Log Files Can

If you are currently not using log files, you may well be using other SEO tools to get you partway to the insight that log files can provide.

Analytics Software

Analytics software like Google Analytics can give you an indication of what pages exist on a website, even if bots aren’t necessarily able to access them.

Analytics platforms also give a lot of detail on user behavior across the website. They can give context as to which pages matter most for commercial goals and which are not performing.

They don’t, however, show information about non-user behavior. In fact, most analytics programs are designed to filter out bot behavior to ensure the data provided reflects human users only.

Although they are useful in determining the journey of users, they do not give any indication of the journey of bots. There is no way to determine which sequence of pages a search bot has visited or how often.

Google Search Console/Bing Webmaster Tools

The search engines’ search consoles will often give an overview of the technical health of a website, like crawl issues encountered and when pages were last crawled. However, crawl stats are aggregated and performance data is sampled for large sites. This means you may not be able to get information on specific pages you are interested in.

They also only give information about their bots. This means it can be difficult to bring bot crawl information together, and indeed to see the behavior of bots from companies that do not offer a tool like a search console.

Website Crawlers

Website crawling software can help with mimicking how a search bot might interact with your site, including what it can technically access and what it can’t. However, they do not show you what the bot actually accesses. They can give information on whether, in theory, a page could be crawled by a search bot, but do not give any real-time or historical data on whether the bot has accessed a page, when, or how frequently.

Website crawlers are also mimicking bot behavior in the conditions you are setting them, not necessarily the conditions the search bots are actually encountering. For example, without log files, it is difficult to determine how search bots navigated a site during a DDoS attack or a server outage.

Why You Might Not Use Log Files

There are many reasons why SEOs might not be using log files already.

Difficulty In Obtaining Them

Oftentimes, log files are not straightforward to get to. You may need to speak with your development team. Depending on whether that team is in-house or not, this may literally mean trying to track down who has access to the log files first.

For teams working agency-side, there is an added complexity of companies needing to transfer potentially sensitive information outside of the organization. Log files can include personally identifiable information, for example, IP addresses. For those subject to rules like GDPR, there may be some concern around sending these files to a third party. There may be a need to sanitize the data before sharing it. This can be a material cost of time and resources that a client may not want to spend simply to share their log files with their SEO agency.

User Interface Needs

Once you have access to log files, it isn’t all smooth sailing from there. You will need to understand what you are looking at. Log files in their raw form are simply text files containing string after string of data.

It isn’t something that is easily parsed. To truly make sense of log files, there is usually a need to invest in a program to help decipher them. These can range in price depending on whether they are programs designed to let you run a file through on an ad-hoc basis, or whether you are connecting your log files to them so they stream into the program continuously.

Storage Requirements

There is also a need to store log files. Alongside being secure for the reasons mentioned above, like GDPR, they can be very difficult to store for long periods due to how quickly they grow in size.

For a large ecommerce website, you might see log files reach hundreds of gigabytes over the course of a month. In those instances, it becomes a technical infrastructure issue to store them. Compressing the files can help with this. However, given that issues with search bots can take several months of data to diagnose, or require comparison over long time periods, these files can start to get too big to store cost-effectively.

Perceived Technical Complexity

Once you have your log files in a decipherable format, cleaned and ready to use, you actually need to know what to do with them.

Many SEOs have a big barrier to using log files simply based on the fact they seem too technical to use. They are, after all, just strings of information about hits on the website. This can feel overwhelming.

Should SEOs Use Log Files?

Yes, if you can.

As mentioned above, there are many reasons why you may not be able to get hold of your log files and transform them into a usable data source. However, once you can, it will open up a whole new level of understanding of the technical health of your website and how bots interact with it.

There will be discoveries made that simply could not be achieved without log file data. The tools you are currently using may well get you part of the way there. They will never give you the full picture, however.

More Resources:

Featured Image: Paulo Bobita/Search Engine Journal

Ask An SEO: Can AI Systems & LLMs Render JavaScript To Read ‘Hidden’ Content? via @sejournal, @HelenPollitt1

For this week’s Ask An SEO, a reader asked:

“Is there any difference between how AI systems handle JavaScript-rendered or interactively hidden content compared to traditional Google indexing? What technical checks can SEOs do to confirm that all page critical information is available to machines?”

This is a great question because beyond the hype of LLM-optimization sits a very real technical challenge: ensuring your content can actually be found and read by the LLMs.

For several years now, SEOs have been fairly encouraged by Googlebot’s improvements in being able to crawl and render JavaScript-heavy pages. However, with the new AI crawlers, this might not be the case.

In this article, we’ll look at the differences between the two crawler types, and how to ensure your critical webpage content is accessible to both.

How Does Googlebot Render JavaScript Content?

Googlebot processes JavaScript in three main stages: crawling, rendering, and indexing. In a basic and simple explanation, this is how each stage works:

Crawling

Googlebot will queue pages to be crawled when it discovers them on the web. Not every page that gets queued will be crawled, however, as Googlebot will check to see if crawling is allowed. For example, it will see if the page is blocked from crawling via a disallow command in the robots.txt.

If the page is not eligible to be crawled, then Googlebot will skip it, forgoing an HTTP request. If a page is eligible to be crawled, it will move to render the content.

Rendering

Googlebot will check if the page is eligible to be indexed by ensuring there are no requests to keep it from the index, for example, via a noindex meta tag. Googlebot will queue the page to be rendered. The rendering may happen within seconds, or it may remain in the queue for a longer period of time. Rendering is a resource-intensive process, and as such, it may not be instantaneous.

In the meantime, the bot will receive the DOM response; this is the content that is rendered before JavaScript is executed. This typically is the page HTML, which will be available as soon as the page is crawled.

Once the JavaScript is executed, Googlebot will receive the fully constructed page, the “browser render.”

Indexing

Eligible pages and information will be stored in the Google index and made available to serve as search results at the point of user query.

How Does Googlebot Handle Interactively Hidden Content?

Not all content is available to users when they first land on a page. For example, you may need to click through tabs to find supplementary content, or expand an accordion to see all of the information.

Googlebot doesn’t have the ability to switch between tabs, or to click open an accordion. So, making sure it can parse all the page’s information is important.

The way to do this is to make sure that the information is contained within the DOM on the first load of the page. Meaning, content may be “hidden from view” on the front end before clicking a button, but it’s not hidden in the code.

Think of it like this: The HTML content is “hidden in a box”; the JavaScript is the key to open the box. If Googlebot has to open the box, it may not see that content straightaway. However, if the server has opened the box before Googlebot requests it, then it should be able to get to that content via the DOM.

How To Improve The Likelihood That Googlebot Will Be Able To Read Your Content

The key to ensuring that content can be parsed by Googlebot is making it accessible without the need for the bot to render the JavaScript. One way of doing this is by forcing the rendering to happen on the server itself.

Server-side rendering is the process by which a webpage is rendered on the server rather than by the browser. This means an HTML file is prepared and sent to the user’s browser (or the search engine bot), and the content of the page is accessible to them without waiting for the JavaScript to load. This is because the server has essentially created a file that has rendered content in it already; the HTML and CSS are accessible immediately. Meanwhile, JavaScript files that are stored on the server can be downloaded by the browser.

This is opposed to client-side rendering, which requires the browser to fetch and compile the JavaScript before content is accessible on the webpage. This is a much lower lift for the server, which is why it is often favored by website developers, but it does mean that bots struggle to see the content on the page without rendering the JavaScript first.

How Do LLM Bots Render JavaScript?

Given what we now know about how Googlebot renders JavaScript, how does that differ from AI bots?

The most important element to understand about the following is that, unlike Googlebot, there is no “one” governing body that represents all the bots that might be encompassed under “LLM bots.” That is, what one bot might be capable of doing won’t necessarily be the standard for all.

The bots that scrape the web to power the knowledge bases of the LLMs are not the same as the bots that visit a page to bring back timely information to a user via a search engine.

And Claude’s bots do not have the same capability as OpenAI’s.

When we are considering how to ensure that AI bots can access our content, we have to cater to the lowest-capability bots.

Less is known about how LLM bots render JavaScript, mainly because, unlike Google, the AI bots are not sharing that information. However, some very smart people have been running tests to identify how each of the main LLM bots handles it.

Back in 2024, Vercel published an investigation into the JavaScript rendering capabilities of the main LLM bots, including OpenAI’s, Anthropic’s, Meta’s, ByteDance’s, and Perplexity’s. According to their study, none of those bots were able to render JavaScript. The only ones that were, were Gemini (leveraging Googlebot’s infrastructure), Applebot, and CommonCrawl’s CCbot.

More recently, Glenn Gabe reconfirmed Vercel’s findings through his own in-depth analysis of how ChatGPT, Perplexity, and Claude handle JavaScript. He also runs through how to test your own website in the LLMs to see how they handle your content.

These are the most well-known bots, from some of the most heavily funded AI companies in this space. It stands to reason that if they are struggling with JavaScript, lesser-funded or more niche ones will be also.

How Do AI Bots Handle Interactively Hidden Content?

Not well. That is, if the interactive content requires some execution of JavaScript, they may struggle to parse it.

To ensure the bots are able to see content hidden behind tabs, or in accordions, it is prudent to ensure the content loads fully in the DOM without the need to execute JavaScript. Human visitors can still interact with the content to reveal it, but the bots won’t need to.

How To Check For JavaScript Rendering Issues

There are two very easy ways to check if Googlebot is able to render all the content on your page:

Check The DOM Through Developer Tools

The DOM (Document Object Model) is an interface for a webpage that represents the HTML page as a series of “nodes” and “objects.” It essentially links a webpage’s HTML source code to JavaScript, which enables the functionality of the webpage to work. In simple terms, think of a webpage as a family tree. Each element on a webpage is a “node” on the tree. So, a header tag

, a paragraph

, and the body of the page itself are all nodes on the family tree.

When a browser loads a webpage, it reads the HTML and turns it into the family tree (the DOM).

How To Check It

I’ll take you through this using Chrome’s Developer Tools as an example.

You can check the DOM of a page by going to your browser. Using Chrome, right-click and select “Inspect.” From there, make sure you’re in the “Elements” tab.

To see if content is visible on your webpage without having to execute JavaScript, you can search for it here. If you find the content fully within the DOM when you first load the page (and don’t interact with it further), then it should be visible to Googlebot and LLM bots.

Use Google Search Console

To check if the content is visible specifically to Googlebot, you can use Google Search Console.

Choose the page you want to test and paste it into the “Inspect any URL” field. Search Console will then take you to another page where you can “Test live URL.” When you test a live page, you will be presented with another screen where you can opt to “View tested page.”

How To Check If An LLM Bot Can See Your Content

As per Glenn Gabe’s experiments, you can ask the LLMs themselves what they can read from a specific webpage. For example, you can prompt them to read the text of an article. They will respond with an explanation if they cannot due to JavaScript.

Viewing The Source HTML

If we are working to the lowest common denominator, it is prudent to assume, at this point, LLMs can’t read content in JavaScript. To make sure that your content is available in the HTML of a webpage so that the bots can definitely access it, be absolutely sure that the content of your page is readable to these bots. Make sure it is in the source HTML. To check this, you can go to Chrome and right click on the page. From the menu, select “View page source.” If you can “find” the text in this code, you know it’s in the source HTML of the page.

What Does This Mean For Your Website?

Essentially, Googlebot has been developed over the years to be much better at handling JavaScript than the newer LLM bots. However, it’s really important to understand that the LLM bots are not trying to crawl and render the web in the same way as Googlebot. Don’t assume that they will ever try to mimic Googlebot’s behavior. Don’t consider them “behind” Googlebot. They are a different beast altogether.

For your website, this means you need to check if your page loads all the pertinent information in the DOM on the first load of the page to satisfy Googlebot’s needs. For the LLM bots, to be very sure the content is available to them, check your static HTML.

More Resources:

Featured Image: Paulo Bobita/Search Engine Journal

Ask an SEO: Is An XML Or HTML Sitemap Better For SEO? via @sejournal, @HelenPollitt1

In this edition of Ask An SEO, we break down a common point of confusion for site owners and technical SEOs:

Do I need both an XML sitemap and an HTML one, and which one is better to use for SEO?

It can be a bit confusing to know whether it’s better to use an XML sitemap or an HTML one for your site. In some instances, neither is needed, and in some, both are helpful. Let’s dive into what they are, what they do, and when to use them.

What Is An XML sitemap?

An XML sitemap is essentially a list of URLs for pages and files on your website that you want the search bots to be able to find and crawl. You can also use the XML sitemap to detail information about the files, like the length of run-time for the video file specified, or the publication date of an article.

It is primarily used for bots. There is little reason why you would want a human visitor to use an XML sitemap. Well, unless they are debugging an SEO issue!

What Is The XML Sitemap Used For?

The purpose of the XML sitemap is to help search bots understand which pages on your website should be crawled, as well as giving them extra information about those pages.

The XML sitemap can help bots identify pages on the site that would otherwise be difficult to find. This can be orphaned pages, those with low internal links, or even pages that have changed recently that you may want to encourage the bots to recrawl.

Best Practices For XML Sitemaps

Most search bots will understand XML sitemaps that follow the sitemaps.org protocol. This protocol defines the necessary location of the XML sitemap on a site, schema it needs to use to be understood by bots, and how to prove ownership of domains in the instance of cross-domain references.

There is typically a limit on the size an XML sitemap can be, and still be parsed by the search bots. This means when building an XML sitemap, you should ensure it is under 50 MB uncompressed, and no more than 50,000 URLs. If your website is larger, you may need multiple XML sitemaps to cover all of the URLs. In that instance, you can use a sitemap index file to help organize your sitemaps into one location.

As the purpose of the XML sitemap is typically to help bots find your crawlable, indexable pages, it is usually necessary to ensure the file references it contains all lead to URLs with 200 server response codes. In most instances, the URLs should be the canonical version, and not contain any crawl or index restrictions.

Things To Be Aware Of With XML Sitemaps

There may be good reasons to go against “best practice” for XML sitemaps. For example, if you are instigating a lot of redirects, you may wish to include the old URLs in an XML sitemap even though they will return a 301 server response code. Adding a new XML sitemap containing those altered URLs can encourage the bots to recrawl them and pick up the redirects sooner than if they were just left to find them via crawling the site. This is especially the case if you have gone to the trouble of removing links to the 301 redirects on the site itself.

What Is An HTML Sitemap?

The HTML sitemap is a set of links to pages within your website. It is usually linked to from somewhere on the site, like the footer, that is easily accessed by users if they are specifically looking for it. However, it doesn’t form part of the main navigation of the site, but more as an accompaniment to it.

What Is An HTML Sitemap Used For?

The idea of the HTML sitemap is to serve as a catch-all for navigation. If a user is struggling to find a page on your site through your main navigation elements, or search, they can go to the HTML sitemap and find links to the most important pages on your site. If your website isn’t that large, you may be able to include links to all of the pages on your site.

The HTML sitemap pulls double duty. Not only does it work as a mega-navigation for humans, but it can also help bots find pages. As bots will follow links on a website (as long as they are followable), it can aid in helping them to find pages that are otherwise not linked to, or are poorly linked to, on the site.

Best Practices For HTML Sitemaps

Unlike the XML sitemap, there is no specific format that an HTML sitemap needs to follow. As the name suggests, it tends to be a simple HTML page that contains hyperlinks to the pages you want users to find through it.

In order to make it usable for bots too, it is important that the links are followable, i.e., they do not have a nofollow attribute on them. It is also prudent to make sure the URLs they link to aren’t disallowed through the robots.txt. It won’t cause you any serious issues if the links aren’t followable for bots; it just stops the sitemap from being useful for bots.

Things To Be Aware Of With HTML Sitemaps

Most users are not going to go to the HTML sitemap as their first port of call on a site. It is important to realize that if a user is going to your HTML sitemap to find a page, it suggests that your primary navigation on the site has failed them. It really should be seen as a last resort to support navigation.

Which Is Better To Use For SEO?

So, which is more important for SEO? Well, neither. That is, it really is dependent on your website and its needs.

For example, a small website with fewer than 20 pages may not have a need for either an XML sitemap or an HTML sitemap. In this instance, if all the pages are linked to well from the main navigation system, the chances are high that users and search bots alike will easily be able to find each of the site’s pages without additional help from sitemaps.

However, if your website has millions of pages, and has a main navigation system that buries links several sub-menus deep, an XML sitemap and an HTML sitemap may be useful.

They both serve different purposes and audiences.

When To Use The XML Sitemap

In practice, having an XML sitemap, or several, can help combat crawl issues. It gives a clear list of all the pages that you want a search bot to crawl and index. An XML sitemap can also be very helpful for debugging crawling issues, as when you upload it to Google Search Console, you will get an alert if there are issues with it or the URLs it contains. It can allow you to narrow in on the indexing status of URLs within the XML sitemap. This can be very useful for large websites that have millions of pages.

Essentially, there isn’t really a reason not to use an XML sitemap, apart from the time and cost of creating and maintaining them. Many content management systems will automatically generate them, which can take away some of the hassle.

Really, if you can have an XML sitemap, you might as well. If, however, it will be too costly or developer-resource intensive, it is not critical if your site is fairly small and the search engines already do a good job of crawling and indexing it.

When To Use The HTML Sitemap

The HTML sitemap is more useful when a website’s navigation isn’t very intuitive, or the search functionality isn’t comprehensive. It serves as a backstop to ensure users can find deeply buried pages. An HTML sitemap is particularly useful for larger sites that have a more complicated internal linking structure. It can also show the relationship between different pages well, depending on the structure of the sitemap. Overall, it is helpful to both users and bots, but is only really needed when the website is suffering from architectural problems or is just exceedingly large.

So, in summary, there is no right or wrong answer to which is more important. It is, however, very dependent on your website. Overall, there’s no harm in including both, but it might not be critical to do so.

More Resources:

Featured Image: Paulo Bobita/Search Engine Journal