How SEOs Can Get Access to Their Server Log Files

4th June 2019Chris Bryant

By Chris Bryant, Search Data Specialist, Botify

Log file analysis for SEO has grown in popularity over the past few years. While not a new topic, the growing complexity of the web has made this a more sought-after tactic than ever before.

If you’re not familiar, log file analysis allows you to see, based on your server logs, how Googlebot crawls your website. In other words, log file analysis lets you see your website how Google sees it.
Log file analysis is important for SEO because it:

• Allows you to see where Google might be spending time on unimportant pages, like duplicate content, while ignoring your important pages or sections.
• Reveals errors, such as 4XX or 5XX errors that could cause Google to spend less time on your website.
• Shows how frequently Googlebot crawls your website, and if that frequency matches your publishing schedule.

There are plenty of SEO use cases for log file analysis that, if you’re reading this article, you’re probably aware of. If you want, you can read more information on the merits of log file analyzing for SEOs.

Most SEO professionals, especially Technical SEOs, don’t need to be convinced of the merits of log file analysis, nor do they take issue with learning how to perform one. In many cases, just getting access to the server logs themselves is the hardest part.

Where are server log files stored and how can I get them?

How you get your log files will first depend on your unique server setup. There are many types, including custom setups, but the most common are:

• Apache
• NGINX
• IIS

Retrieving log files can be cumbersome, even if you’re the person who configured the server and set up the website from scratch. As an example, to retrieve log files from Apache, you need to:

1. Log into your remote server (typically a linux environment)
2. View your Apache configuration to see where the log file is physically located on the machine. It may be in a default location, but some versions of Apache have CustomLog directives in your configuration (see their manual for more information)
3. Use SFTP or some other protocol to download your log files.

Even with all of that, you may still run into trouble. For example:

• In some configurations, Apache may not be storing a meaningful amount of data, meaning log files need to be downloaded frequently and then aggregated on your local computer.
• Depending on how your log files are exported, there may be a ton of extra data that an SEO wouldn’t need.
• You may miss a download, meaning you lose important metrics.
• Excel files become too big and unusable.

In some cases, you can download log files through your web hosting providers admin panel (like cPanel), but in any organization larger than a few people, an SEO may not have direct access to log files, and may get push back from DevOps or Engineering teams on getting access or having log files delivered directly to them.

“Um, this is an 800mb file. What are you going to do with it?”

Sound familiar?

At best, they may agree to give you three days’ worth of data — not incredibly helpful for your purposes.

Getting your log file data for SEO: What to do instead

So we all agree that doesn’t sound ideal, right?

Right.

Here’s what to do instead.

Instead of bugging your DevOps team for your log files to no avail, instead of trying to manage log files yourself and crashing Excel, and instead of having to manually parse through millions of cells worth of data, you can have your log files automatically fed into a usable interface made specifically for SEOs.

We’re talking about Botify’s own Log Analyzer, and it works like this:

• Sign up for Botify
• Our engineering support team connects with your DevOps team (or whoever controls your server log files)
• We set them up to feed directly into your Botify platform
• Boom! All your relevant log file data in an easy-to-use interface

We know you’re smart enough to secure your log files and view them in their raw form, but when you could be spending your time gleaning insights from your data, why waste your time just trying to access them?

The Botify Log Analyzer not only completely lifts the burden of accessing your log files off your shoulders, but it also puts all your data into an interface that’s easy to make sense of. It also enables SEOs to view their log files by date, which is useful for diagnosing issues.

We want you to be able to spend your time on analysis, not on busy work.

Want to see how it works for yourself? Book a demo with us! We’re looking forward to showing you all that Botify has to offer.