Do Slow Page Load Times Negatively Impact How Google Crawls Your Site?

31st July 2019Kyle Blanchette

It’s widely accepted in the SEO community that page load times impact how Google crawls your site — as load time increases, crawl ratio decreases. Like most accepted truths though, we wanted to see if the data actually backed that up.

Botify analyzed 413 million pages crawled by Botify and 6 billion Googlebot requests to learn how Google crawls the web, and shared our findings at SMX Paris 2018. When we looked at the data to see whether load time impacted crawl, we considered it from a web crawler’s point of view, analyzing time to first byte (web server responsiveness) and the time to download the HTML document.

What we found was that, on sites with less than 10,000 pages, load times have a pretty limited effect on Google’s crawl. On larger sites, however, load time’s impact on crawl is huge.

Why does it matter how Google crawls my site?

To fully appreciate these findings, you have to understand concepts like crawl ratio, crawl frequency, and how they impact your site as a whole.

What is crawl ratio?

Crawl ratio is the percentage of indexable pages on your website crawled by Google in the last 30 days. Google doesn’t have infinite resources, so they have a crawl budget that defines how long they’ll spend on a website before leaving. If Googlebot isn’t visiting some of your pages, those pages won’t be indexed, and if a page isn’t indexed, it can’t show up in search results when people are looking for it — bad news if those pages are important for your business.

What is crawl frequency?

Crawl frequency is the average number of times a URL was crawled by Google in the last 30 days. Google’s crawl process is algorithmic, meaning computer programs determine how often they’ll crawl. If you update your content often, you’ll want Google to crawl your pages often as well. Infrequent crawls on a website that’s constantly publishing new content can pose a problem, because it would lead to a delay between the time of publish and the time searchers could actually find that new content.

Together, crawl ratio and crawl frequency answer the question, “How many pages of the website is Google crawling, and how often?”

The data shows that Google simply has an easier time crawling when your pages are fast, so if you have a website with more than 10,000 URLs, you need to optimize for page load time or else risk some of your important content being missed by Google, and consequently, by searchers.

Other reasons to care about page load times

So now we have the evidence to support that load times impact Google’s crawl, and we know that crawling impacts which of our content gets indexed and found by searchers, but are there any other reasons to care about load times?

Absolutely!

Google uses speed as a ranking factor on both mobile and desktop. High-quality content that matches the query intent is still a prime ranking factor, so Google says that slow pages may still rank well if their content is great. However, it’s important to remember that speed can change the way your content performs in search.

It can also be easy as SEOs to focus on how Google is interacting with our website and forget the impact on the end user, but page speed is a critical component of user experience. If a page doesn’t load within 3 seconds, most people will get frustrated and leave.

Because a few seconds can mean the difference between making or losing the conversion, page load times can have massive implications for revenue. That’s led to some organizations (typically larger enterprises) to establish performance teams whose entire jobs are dedicated to optimizing for things like page load times and user experience.

If you don’t have a performance team and you’re trying to get developer or executive buy-in for a page speed initiative, explaining the impact on user experience and revenue can be more effective than solely emphasizing load time’s impact on Google’s crawl.

So what are the implications of page speed on search?

1. Crawling. Slow pages can mean less pages crawled by Googlebot.
2. Ranking. Slow pages may rank worse on desktop and mobile.
3. User Experience. Slow pages can lead to higher bounce rates and lower conversions.

My pages are slow. Where should I start?

Performance is one of the biggest issues we run into but one of the hardest to address. When you have a site with potentially millions of pages, tackling this issue can seem overwhelming and even impossible. That’s why we recommend drilling down and looking at your site in smaller segments.

For example, you may not have performance issues across your entire site. They may be exclusive to a section of your site that’s hosted on a different platform, like a blog that’s on WordPress, your international pages, or only your product pages. Segmentation adds context, making it clearer what you need to work on.

Once you’ve identified your worst-performing segments, use a tool like Google’s PageSpeed Insights to find the biggest culprits of slow load times.

Some of the biggest contributors to slow page load times are:

• Images that aren’t compressed or sized properly
• JavaScript that isn’t loaded asynchronously or deferred
• You’re not using a content delivery network (CDN)
• Unnecessary plugins or tracking scripts

However, it’s important to remember that there are countless causes of slow page speeds, and you should always test your own pages to see what your unique issues are. Google even has a tutorial that shows you how to use Chrome DevTools to make your pages load faster.

It’s important to remember that, when it comes to the web, everything impacts each other. You may make page load improvements with the goal of improving crawl ratio and frequency, but the effects don’t stop there.

Improving page load times improves user experience, which can conversely improve conversions. Getting more of your important pages crawled can also lead to more pages indexed, and consequently more traffic and conversions. Because of the compounding effects of every website change, saying you’re making improvements “for SEO” doesn’t really do it justice.

As with any other change, page load times can drastically affect how both search engines and your real human visitors engage with your brand. If you’re curious how load times might be impacting KPIs like crawling, traffic, and conversions, request a demo with us!