The 5 Biggest XML Sitemap Mistakes to Avoid and Boost Your SEO

15th May 2019Will Lau

By Will Lau, Sr. Search Data Specialist, Botify

XML Sitemaps are a huge help to your website’s SEO–that is, IF you do sitemaps right.

XML sitemaps help search engines like Google crawl your pages more intelligently. They tell crawlers where your important pages are located and even provide information such as when your pages were last updated. Consequently, mistakes in XML sitemaps can send confusing signals to Google. These mistakes might cause Google to miss your important pages or waste time crawling unimportant pages.

Sitemaps can be especially important on JavaScript-dependent websites. In a study Botify conducted with Merkle, we found that Googlebot’s difficulties crawling some JavaScript websites cause it to rely more heavily on sitemaps to get a full picture of the website.

For all these reasons, sitemaps are critical to your site performing well in organic search, which is why we’ve compiled some of the most common XML sitemap mistakes and steps you can take to avoid them.

1. Listing non-indexable pages

Google warns us to only include consistent, fully-qualified URLs in our sitemaps. That means we need to avoid sending Google to non-indexable pages that send confusing signals as to which pages we want Google to crawl and index.

If you’re not familiar with Botify, we categorize pages into “indexable” and “non-indexable” categories. Indexable pages are those pages that you want Google to index, judging by your page’s directives.

In other words, indexable URLs are any URLs on your website that:

• Respond with an HTTP 200 OK status code (it’s not 404ing, redirected, etc.)
• Are the preferred versions of your pages, and not the alternates (ex: the canonical versions of your pages)
• Are HTML content type pages
• Do not contain a meta No-Index tag

As you can see from this list, there are lots of directives you can give that would cause Google not to include pages in its index. Without that indexable vs. non-indexable designation, you’d likely have to crawl all your URLs and rule each of these out manually before arriving at a true list of pages you want to be indexed.

In Botify, you can just go to the URL explorer and filter to view all indexable pages.

How does this help you with your XML sitemap? We’re glad you asked!

Just use our handy “Export XML Sitemaps” functionality and automatically export your site’s full list of indexable URLs into XML sitemap format.

2. Forgetting to update your sitemap after a migration

In the frenzy of a site migration, it can be easy to forget even basic things like updating your XML sitemap.

Make sure that when you move from your old site to your new site, you update your XML sitemap so that Google has an accurate list of URLs.

For example, make sure your robots.txt file references your new XML sitemap(s) and that you submit your new version to Google in Google Search Console.

3. Omitting important pages

Are some of your important pages missing from your XML sitemap? While your XML sitemap isn’t the only way that Google finds your pages (they find them through links on your site and other sites as well), it’s still a good idea for your XML sitemap to represent all the URLs you want Google to index.

In Botify, you can utilize segments to see if your strategic pages are included in your sitemap(s):

…

4. Forgetting to submit your sitemap to Google

Like we mentioned already, submitting your sitemap to Google isn’t the only way that Google can find your pages, nor is it a guarantee that Google will crawl and index your pages. Still, submitting your sitemap to Google is an important part of ensuring that you cover your bases when it comes to getting your important pages crawled and indexed.

In Google’s own words, “If your site’s pages are properly linked, our web crawlers can usually discover most of your site. Even so, a sitemap can improve the crawling of your site.”

Google says that submitting a sitemap to them is especially valuable in the following situations:

• You have a large site
• Your site doesn’t have great internal linking
• Your site doesn’t have many external links to it

5. Creating an XML sitemap manually

Google says that you can create a sitemap either manually or by using a third-party tool. We recommend using a third party tool like Botify’s XML Sitemap Generator as this will save you loads of time.

The Botify XML Sitemap Generator eliminates the grunt work in the sitemap creation process by automatically generating a proper XML sitemap from your own customized list of URLs. For example, you can instantly generate XML sitemaps specifically for URLs that:

• Are indexable, but not getting organic visits
• Are indexable, but Google isn’t crawling
• Are indexable, but don’t have many internal links

This will automatically allow you to export an XML sitemap in the proper UTF-8 format. It also follows Google’s XML sitemap best practices by automatically breaking up sitemaps into no more than 50,000 rows each.

Generate XML sitemaps and eliminate errors with Botify

At Botify, we’re all about saving SEOs time. That’s why we aim to automate as much as possible so you can spend more time on meaningful analysis.

While there are plenty of SEO software suites out there, we think ours is pretty special in that it allows you to unify your data in new and interesting ways, such as what you saw with creating custom XML sitemaps for different use cases.

If you’d like to see what we’re all about, and how we might be able to save you time, book your demo with us today!