Proof is in the HTTP Pudding

25th August 2016Elizabeth

How are you optimizing your site for increased organic traffic?

You can update your content, add new pages, and build a complex web of internal linking, but you could still be missing a huge area of opportunity for optimization.

What’s your status: HTTP Status Codes & Compliance

Compliance is everything. When a crawler like Googlebot gets to your site, what does it find? Does it request each page and your server seamlessly returns a 200 OK status code? What about 3XX redirects or, even worse, 4XX and 5XX errors?

What is compliance

There are 4 basic criteria a URL should meet to be considered “indexable”:

1. The page responds with an HTTP 200 (OK) status code: No redirect (3xx) or error (4xx, 5xx).
2. The page has no canonical tag, or a canonical tag to itself. The page cannot be a non-primary version of duplicate content managed via canonical tags: the page does not include a canonical tag to another URL, saying that the main version of its content is elsewhere.
3. The page has an HTML content type.
4. The page does not include any Noindex meta tag, which prevents it from being indexed by search engines.

Non-indexable pages can be a huge drain on your potential SEO traffic generation. HTTP status codes are an excellent indicator of your site’s compliance, as HTTP errors result in negative impacts on both user experience for your potential site visitors and customers, as well as for your ranking with Google and its ilk.

In fact, for large and complex websites, non-indexable pages could represent an enormous proportion of the site, resulting in wasted crawl budget and negative impacts on site perception by picky search engines like Google.

And beyond overt errors, redirects are a whole other can of worms. Although Google recently came out and officially stated that 301 or other 3XX redirects no longer cost you PageRank, there are still important SEO reasons to fix these non-indexable pages.

Fixing bad HTTP status codes

Fixing and removing HTTP errors like 4XX and 5XX status codes is critical to your site’s success, and should be one of your first steps in SEO.

When search engine crawlers like Googlebot encounter a page that returns an error, it is wasting Google’s precious (and limited) crawl budget resources when it could be crawling other, more important pages. It also sends a signal to Google that maybe your site isn’t so high quality after all. As these errors add up, your site’s value goes down.

Prioritizing SEO updates: Reasons for non-compliance

What should you fix right now, right this second, and what can wait a bit? Plain errors – especially on pages with important content – require immediate attention to stop losing out on the value of those pages and the bad user experience they’re causing. Other reasons for non-compliance can vary, and fall elsewhere in your to-do list depending on the importance of those pages and the cause of the HTTP status.

Use this guide to prioritize your non-indexable pages:

• Error HTTP Status Code: Always correct 4XX and 5XX pages. Especially those errors that are shallower in depth, as the closer to the homepage an error is, the greater the impact that page error can have on Googlebot’s crawl.

Take action if there is a large number or proportion of these pages:

• Canonical Not Equal. Too many of these pages suggest that there’s a lot of overlap in your pages, whether subtle or complete [duplicate content](https://www.botify.com/blog/content-quality-analysis-SEO-how/). Crawlers don’t know pages are duplicated until they explore them, which means they’re wasting time exploring pages that aren’t adding SEO value.
• Meta Noindex pages are not good in large quantities. Links from these pages are less likely to be explored by search engines, and without links noindex pages just waste crawl time that could be spent elsewhere on SEO-valuable pages.
• Bad Content Type. If the content type is incorrect, these pages are being crawled but not returning the value they should. Or, it’s correct but possibly unworthy of being crawled as a .pdf, links to images, or other content types. Large volumes in either case are, of course, wasting significant crawl resources.

Do these critical status code issues sound familiar? Find out how to correct HTTP errors here, according to the cause of non-compliance.

Regardless of the reason behind non-compliance, it’s important to address the causes and remove 3XX redirects, 4XX and 5XX errors in order to make the most of Google’s limited time exploring your site so your most valuable pages get seen quickly and efficiently.

Why? The SEO Proof is in the (Organic Traffic) Pudding

When your pages aren’t returning 200 OK status codes, it’s time to dig into the reasons why – and prioritize their improvement. Non-indexable pages can have significant impacts on your SEO by lowering the apparent quality of your website in the eyes of search engines and hindering efficient exploration of indexable, important pages.

When you make site changes to fix or remove HTTP status code errors or redirects, you’ll soon see real results in the form of more pages crawled, more valuable pages crawled, and more organic traffic flowing into your site. And, presumably, you’ll see increased revenue as a result.

The proof, as they say, will be in the pudding.

Don’t believe us? Try Botify and see for yourself how you can drive traffic and revenue, starting with HTTP status codes.