Why Do SEOs Need an SEO Log File Analyzer?

12th October 2017Kevin Bartley

How can you generate a large amount of new traffic and revenue without creating any new content?

Sometimes that content is right under your nose. Google’s limited crawling resources only crawl a portion of enterprise websites. By prioritizing strategic, traffic driving pages in your crawl budget, your site can gain serious SEO ROI, including increased revenue. To gain this kind of broad insight, you’ll need a tool that offers real visibility into the crawl.

That’s where an SEO log file analyzer comes in.

SEOs have always liked digging into log files to find advanced SEO optimizations. The Botify founders share that obsession with this powerful source of data. That’s why Botify was the first company to offer a SaaS solution for SEO log file analysis. Our company was, in fact, started with our SEO log analyzer.

Now we’ve become the first company to capture the entire SEO process, from crawling, to indexing, to ranking, in a single powerful enterprise platform.

Though we’ve greatly expanded the capabilities of our solutions, the SEO log file analyzer is still at the core of our platform. But you might be asking, why do SEOs need an SEO log file analyzer in the first place?

Search Engines Don’t Have a Perfect Understanding of Your Content, Help Them!

Although search bots are powerful, they can’t, and they won’t, crawl or discover every piece of content on the web. Even Google’s seemingly all-knowing bots can miss wide swaths of large websites. On an average 30 day period, we at Botify find that around 40% of pages all enterprise websites remain uncrawled.

An SEO log file analyzer can help you understand why portions of your website are not getting crawled. Discover the crawl ratio of your site by reviewing search engine crawl of your entire site and pairing crawled and uncrawled pages side by side.

The crawl ratio can vary widely depending on website performance. The first thing you want to do is identify how many of your crawled pages are non-indexable (not eligible to be served in search).

As shown above, if your site’s crawled pages by search engines are 41.4% indexable and 58.6% non-indexable, that means a wide portion of your crawl budget is being spent on useless pages. That’s dead weight yielding no organic traffic.

You can isolate these non-indexable pages and optimize them, hide them, or remove them to save crawl budget. A good SEO log file analyzer will also let you segment your pages so you can understand crawl performance by page type (products, category, articles, etc.)

Review uncrawled pages to see if those are less valuable pages or if you do, in fact, want those pages to be crawled. An SEO log file analyzer allows you to isolate strategic pages that aren’t getting crawled and understand the technical reasons why they’re being ignored by the search engines.

An SEO log file analyzer can also offer important metrics about crawl frequency. A higher crawl frequency signifies that Google is finding and understanding the content on your website more frequently, which is a primary aim of crawl budget optimization. The more often your pages are being crawled, the more likely it is that they can drive traffic.

The charts below signify crawl frequency. On the left chart, 43% of pages have never been crawled during the month, and 49% have been crawled only six days out of the thirty analyzed. Compare this to the chart on the right. The conclusion is much different: 98% of the pages have been crawled at least 24 days (or higher) out of 30.

With an SEO log analyzer, you can understand and chart the relationship between crawl frequency, crawl budget, and traffic fluctuations.

Use SEO Log Analysis to Make Strategic Projects a Success

As an SEO, you probably have some prioritized projects, such as ensuring content investment, setting up an HTTPS migration, and initiating your AMP strategy. Consider these scenarios as examples of how to make a strategic project successful with an SEO log file analyzer.

New Content Discovery

Consider a company that’s prepared to launch a new set of products. This company launches a series of new product pages, and wants to know how many of these pages have been crawled.

With an SEO log file analyzer, this company can group its new product pages through segmentation, and see exactly when these specific pages have been crawled. In a matter of days, the company can be 100% certain that this new strategic content has been discovered by Google. The company can then tie these crawled pages back to the resulting visits.

As you see, the rise in visits corresponds to the newly crawled pages. This way, the company can verify exactly when pages are crawled, and when they start to generate organic visits. This is crucial to ensure the ROI of content.

HTTPS Migration

You can use crawl metrics to monitor successful implementation of scenarios like HTTPS migration. Break HTTP and HTTPS down by segment. On the left, you can see that within a few days of starting the migration, the crawl volume of HTTPS pages increased. You can also see visits to HTTPS take over after a few days, confirming that the migration was successful.

AMP Strategy

With an SEO log file analyzer, you can understand the crucial distinction between mobile and desktop search bot activity. See if your AMP pages are being crawled, for example, as in the charts above. Compare how Google’s different crawlers are discovering the content.

Compare the way Google Smartphone is crawling the AMP pages to how Googlebot is crawling AMP pages as in the example above. Optimize pages for particular user agents. This scenario is explained in more detail in our recent mobile webinar.

Log Files Help you Improve Search Engine Perception, Ranking, and Revenue

The insights of an SEO log analyzer will ultimately lead to higher traffic and revenue. The goal of optimizing your crawl budget is to get more and better pages crawled by Google, which will lead to improved rankings and increased organic visitors, some of whom will become customers.

For example, if you do improve your crawl ratio, what effect does it have on traffic? Consider this example of a publishing website. Back in March 2016, the daily visits looked like this:

A year and a half later, in September 2017, the daily visits looked like this:

The visits improved by roughly 300,000 per day, and almost 500,000 on peak days. That represents about a 60% increase in organic traffic since March 2016. How did they achieve this huge boost?

Using an SEO log file analyzer, the site optimized crawl budget and added about 60,000 strategic pages to Google’s crawl. This led to a massive jump in organic traffic.

With Botify Log Analyzer, alongside our powerful crawler Botify Analytics, you can go further than these overhead insights, and examine individual URLs for inlinks, delay, content quality, depth, and 350+ technical SEO markers that affect crawling, and eventually, ranking and traffic.

SEO Log Analyzer: Join the average Botify customer, improve your audience by 15%

We’re happy to see that, following Botify’s innovations, many SEO agencies and tools are now offering log file analysis. Botify Log Analyzer has paved the way for these new solutions, and it’s positive for the industry that more SEOs are becoming obsessed with their crawl budgets.

SEO log file analysis is obviously not the sole focus of SEO. The highest priorities for an SEO should always be developing high quality mobile and desktop websites, in addition to great content, and significant interaction/popularity with users.

However, SEO log file analyzers are key for gaining high ranking positions, because these tools give you a competitive advantage in understanding how exactly search engines do or don’t appreciate your pages.

With this level of insight, SEOs don’t need to fly blind any more. Using an SEO log file analyzer, you can take control over search engine crawling and base your search engine optimization on data, and immediately communicate the SEO impact to any team.

Are you ready to use the power of SEO log file analysis? Watch our Botify Log Analyzer webinar (in English or book a demo of our solution to see what Botify Log Analyzer and the entire Botify platform can do for you.