Mobile-First Index: A Data-Driven Analysis & Discussion

13th December 2018Kevin Bartley

As more and more sites enter the mobile-first index, the demand for data-driven analysis of the rollout has never been greater.

That’s why we initiated our latest deep dive study, an illumination of the trends and behaviors that are currently governing the mobile-first index.

For the study, we analyzed 190 unique domains, 495 million URLs, and 1.52 billion Googlebot requests, using what remains the largest dataset yet collated for the purposes of assessing the mobile-first index.

The study was similar in design to How Does Google Crawl the Web?, another major report we released this year

What did this latest study on the mobile-first index find? Read on to discover some of the key findings!

What Sites Are Entering the Mobile-First Index – And Why?

By June 2018, only 9.6% of the studied websites had switched over to the mobile-first index.

But according to our research, 34% of the enterprise websites we studied had switched over to the mobile-first index by October 2018.

As the chart implies, the rollout is happening progressively, rather than all at once. That means certain sites will enter the mobile-first index before others.

One of our aims of the study was to uncover what sites Google is prioritizing to enter the mobile-first index right now. In this regard, the data showed that smaller websites are more likely to enter the mobile-first index.

The blue in the graph above represent the percentage of sites that have switched to the mobile-first index for a given site size. As shown, larger sites are less likely to enter the mobile-first index, at least for now.

From a risk assessment point of view, this makes sense, because smaller websites should be easier to move for both Google and the site stakeholders.

Our data also showed that websites with higher crawl ratios are prioritized to enter the mobile-first index. Crawl ratio – derived from SEO log file analysis – represents the percentage of indexable page) crawled by Google in a 30 day period (June – July 2018).

Familiarity with, and comprehension of, a given website both appear to enter Google’s calculus when marking the domain for the transition. In terms of the industry type, publishers are more likely to enter the mobile-first index, rather than industries such as retailers and classifieds.

Potentially due to fresh content, publishers are crawled by Google more often than retailers and classifieds. This leads to higher crawl ratios. Does that explain the high number of publishers in the mobile-first index?

Despite these findings, there were many other key indicators that appeared to have no impact on entering in the mobile-first index, including load time, content size, and visits, among many others.

How Will a Transition to the Mobile-First Index Impact Your Site?

Of course, SEOs want to know how the mobile-first index will impact their sites. Our study illuminated this issue by examining Googlebot’s behavior in granular detail.

We measured the crawl volume, crawl ratio, and performance indicators before/after each site switched over to the mobile-first index to get an average gauge of how these indicators are affected.

Our study also highlighted specific use cases with actual data and real results from some of our biggest customers.

The graph above shows the number of crawls for specific Googlebot user agents over a thirty day period. As you can see, the red bars (which represent Googlebot for Desktop) disappear as the blue bars (Googlebot for Smartphone) become ascendant.

The smooth transition is sealed by the consistency in the crawl volume. Google still crawls the same number of URLs, it’s just doing so with the Smartphone user agent instead of the Desktop user agent.

Now here’s the opposite – a not so smooth transition to the mobile-first index:

In this use case, the switch to the mobile-first index is obvious. Notice the blue spike that represents a significant increase in the crawl volume for Googlebot for Smartphone. The moment of transition is clear.

The problem is that the number of pages crawled actually decreases after the switch to the mobile-first index. This means certain pages aren’t entering search – a big issue that signifies the site wasn’t ready for the mobile-first index.

Our study compared all these kinds of scenarios side-by-side, highlighting the specific optimizations SEOs can make to succeed in the mobile-first index.

Read the report and sign up for the upcoming webinar now to learn the secrets of the mobile-first index!