WordPress Robots.txt: What Should You Include? via @sejournal, @alexmoss
The humble robots.txt file often sits quietly in the background of a WordPress site, but the default is somewhat basic out of the box and, of course, doesn’t contribute towards any customized directives you may want to adopt.
No more intro needed – let’s dive right into what else you can include to improve it.
(A small note to add: This post is only useful for WordPress installations on the root directory of a domain or subdomain only, e.g., domain.com or example.domain.com. )
Where Exactly Is The WordPress Robots.txt File?
By default, WordPress generates a virtual robots.txt file. You can see it by visiting /robots.txt of your install, for example:
https://yoursite.com/robots.txt
This default file exists only in memory and isn’t represented by a file on your server.
If you want to use a custom robots.txt file, all you have to do is upload one to the root folder of the install.
You can do this either by using an FTP application or a plugin, such as Yoast SEO (SEO → Tools → File Editor), that includes a robots.txt editor that you can access within the WordPress admin area.
The Default WordPress Robots.txt (And Why It’s Not Enough)
If you don’t manually create a robots.txt file, WordPress’ default output looks like this:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php
While this is safe, it’s not optimal. Let’s go further.
Always Include Your XML Sitemap(s)
Make sure that all XML sitemaps are explicitly listed, as this helps search engines discover all relevant URLs.
Sitemap: https://example.com/sitemap_index.xml Sitemap: https://example.com/sitemap2.xml
Some Things Not To Block
There are now dated suggestions to disallow some core WordPress directories like /wp-includes/, /wp-content/plugins/, or even /wp-content/uploads/. Don’t!
Here’s why you shouldn’t block them:
- Google is smart enough to ignore irrelevant files. Blocking CSS and JavaScript can hurt renderability and cause indexing issues.
- You may unintentionally block valuable images/videos/other media, especially those loaded from /wp-content/uploads/, which contains all uploaded media that you definitely want crawled.
Instead, let crawlers fetch the CSS, JavaScript, and images they need for proper rendering.
Managing Staging Sites
It’s advisable to ensure that staging sites are not crawled for both SEO and general security purposes.
I always advise to disallow the entire site.
You should still use the noindex meta tag, but to ensure another layer is covered, it’s still advisable to do both.
If you navigate to Settings > Reading, you can tick the option “Discourage search engines from indexing this site,” which does the following in the robots.txt file (or you can add this in yourself).
User-agent: * Disallow: /
Google may still index pages if it discovers links elsewhere (usually caused by calls to staging from production when migration isn’t perfect).
Important: When you move to production, ensure you double-check this setting again to ensure that you revert any disallowing or noindexing.
Clean Up Some Non-Essential Core WordPress Paths
Not everything should be blocked, but many default paths add no SEO value, such as the below:
Disallow: /trackback/ Disallow: /comments/feed/ Disallow: */feed/ Disallow: */embed/ Disallow: /cgi-bin/ Disallow: /wp-login.php Disallow: /wp-json/
Disallow Specific Query Parameters
Sometimes, you’ll want to stop search engines from crawling URLs with known low-value query parameters, like tracking parameters, comment responses, or print versions.
Here’s an example:
User-agent: * Disallow: /*?replytocom= Disallow: /*?print=
You can use Google Search Console’s URL Parameters tool to monitor parameter-driven indexing patterns and decide if additional disallows are worthy of adding.
Disallowing Low-Value Taxonomies And SERPs
If your WordPress site includes tag archives or internal search results pages that offer no added value, you can block them too:
User-agent: * Disallow: /tag/ Disallow: /page/ Disallow: /?s=
As always, weigh this against your specific content strategy.
If you use tag taxonomy pages as part of content you want indexed and crawled, then ignore this, but generally, they don’t add any benefits.
Also, make sure your internal linking structure supports your decision and minimizes any internal linking to areas you have no intention of indexing or crawling.
Monitor On Crawl Stats
Once your robots.txt is in place, monitor crawl stats via Google Search Console:
- Look at Crawl Stats under Settings to see if bots are wasting resources.
- Use the URL Inspection Tool to confirm whether a blocked URL is indexed or not.
- Check Sitemaps and make sure they only reference pages you actually want crawled and indexed.
In addition, some server management tools, such as Plesk, cPanel, and Cloudflare, can provide extremely detailed crawl statistics beyond Google.
Lastly, use Screaming Frog’s configuration override to simulate changes and revisit Yoast SEO’s crawl optimization features, some of which solve the above.
Final Thoughts
While WordPress is a great CMS, it isn’t set up with the most ideal default robots.txt or set up with crawl optimization in mind.
Just a few lines of code and less than 30 minutes of your time can save you thousands of unnecessary crawl requests to your site that aren’t worthy of being identified at all, as well as securing a potential scaling issue in the future.
More Resources:
Featured Image: sklyareek/Shutterstock
