Enterprise SEO

The Beginner’s Guide To Botify

By The Botify Team

Trying to find the right SEO software or tools can be overwhelming. Plus, having too many tools can be a problem in itself. That’s where Botify comes in.

If you’re new to Botify, we’re here to help! We wrote this article to introduce you to who Botify is as a company, what problems we’re solving for enterprise brands, and how our platform works.

Click below to jump to a section:

A little about Botify the company

Founded in 2011 by Adrien Menard, Thomas Grange, and Stan Chauvin, the Botify team has been dedicated to anticipating what’s next in the SEO industry and creating a scalable, solutions-oriented platform to maximize organic search traffic and revenue for enterprise brands. As we’ve grown, we’ve been fortunate enough to do things like gain $20M in Series B Funding, open an office in Seattle (our fourth location!), and expand our team.

As a company, we believe in the power of SEO for driving growth and revenue when it’s properly prioritized and resourced. In fact, we recently commissioned Forrester Consulting to conduct a study on just that! To learn more, you can read the research or watch the video.

But let’s take a step back. One of the most important things you should know about Botify is that we take a “full funnel” approach to SEO, meaning that you need to look at every phase of the search process (crawling, rendering, indexing, ranking, conversions) in order to really understand what improvements you need to make to perform better in search.

Screen Shot 2019-12-05 at 10.11.06 AM

After all, search is a process with a clear beginning and end — if search engines don’t crawl your site, your pages can’t rank in the index where they can earn traffic and revenue for your business. We look at everything through that lens, and it’s the foundation upon which we built our platform. We call it the Botify Methodology.

With that foundation in mind, let’s take a look at the problems we’re solving.

What problem is Botify solving for enterprise businesses?

The fact is, Google crawls less than half of an enterprise website’s pages – that’s significant!

Screen Shot 2019-12-05 at 10.11.58 AM

From the Botify study “How does Google crawl the web?

If search engines aren’t crawling your website, your pages can’t be indexed (and they definitely won’t rank). For enterprise websites, that means there’s a ton of potential for improving traffic and revenue simply by getting search engines to find your important content.

This is where Botify solves a huge problem for large, enterprise websites.

Botify works with organizations that count on their website(s) for a substantial portion of their revenue. In many cases, that translates into working with e-commerce, publishers, classifieds, and travel organizations.

Whether you’re a technical SEO responsible for identifying indexing issues, a content marketer who needs to analyze content for quality, or an executive who’s responsible for driving traffic to and revenue from your organization’s website, Botify can help!

Here’s a run-down of how Botify works

The Botify platform began as a comprehensive search analytics platform, and at its core was our cloud-based crawler that can get through even the biggest sites at a rate of 250 URLs per second. While knowing virtually every detail about your site is critical (from how many links to redirected URLs you have, to the average size and uniqueness of your content), we knew the most actionable SEO insights come when you unify your data.

That’s why, on top of our crawler, we layer on your log file data, your query data, and your website analytics data.

Screen Shot 2019-12-05 at 10.12.51 AM

Botify allows you to see your website at every phase of the search process — from crawling to conversions — and how each part impacts another (ex: how crawl impacts ranking, how content quality impacts crawl frequency, etc.). You can layer 1,000+ data points for any URL on your website to identify exactly what’s impacting your pages.

With so much data at their disposal, our customers are able to use Botify for things like analyzing their website architecture for SEO, finding and fixing indexing issues, identifying trending keywords, and more — the use cases really are endless!

Botify Intelligence & Botify Activation

But with great quantities of data comes great responsibility — you have to take the time to peruse your data to pull out the highest-priority insights. That’s why the next iteration of Botify is all about intelligent, prescriptive actions and automation.

Botify Intelligence and Botify Activation, which we announced in October 2019, take your search data to the next level, solving for pain points like “With all this data, I’m not sure where to start,” “Critical pages were blocked by robots.txt and I wasn’t aware until our traffic suffered,” and “I can never seem to get my SEO projects implemented because my dev team doesn’t have the resources.”

Intelligent alerts and automation is only possible because of the completeness and trustworthiness of the data that’s made possible through Botify Analytics, so let’s take a quick dive into what makes Botify Analytics tick.

Botify Analytics: What’s inside?

Crawl Data

With Botify Analytics, it all starts with a crawl. The data we retrieve from each crawl helps our customers evaluate the structure and content of every page on their website — from mobile to desktop — just as a search engine would. Since all of your SEO data lives in Botify, you can easily keep track of changes in your crawl data over time.

Not only does Botify crawl HTML at a rate of 250 URLs per second, it can also fully crawl and render your JavaScript (using the same rendering engine as Googlebot), at a rate of 100 URLs per second, critical for JavaScript SEO audits.

Log File Data

The log file analysis component of Botify Analytics enables you see how search engine bots are crawling your site. You can feed your server log files into a dashboard built specifically for SEOs to overlay them with your crawl data. This allows you to see which pages Google is ignoring and which pages they’re visiting, enabling you to detect crawl budget issues. It’s essential to use this data in order to direct bots to your most critical pages — your top converters and top revenue generators.

Screen Shot 2019-12-05 at 10.13.34 AM

Keyword & SERP Performance Data

Through the Google Search Console API on our platform, you’ll be able to see even more data than in the GSC performance dashboard. Our customers generally use this feature to get all of their query data, such as what queries are driving impressions and clicks. The keywords dashboard also displays other keyword insights, such as where your audience is searching from, what devices they’re using, missed clicks (i.e. where you’re getting impressions but not clicks), and keywords that are trending up or down with your audience. Additionally, we don’t scrape the SERPs to get the data; we pull directly from GSC.

Web Analytics Integration

Botify Analytics works hand in hand with Google Analytics, Adobe Analytics, and also AT Internet. We can connect with your site’s analytics account to bring in valuable visits and conversion data to overlay with your Botify Analytics data (crawl data, log file data, and keyword data). This helps you see real trends in how your site’s technical issues are impacting traffic and engagement.

What are the benefits of unifying your SEO data?

When utilizing all of Botify together, you can uncover big insights that you’ve never been able to see before. Here are some examples:

  • Layering title tag data (from a crawl) with your click data (from keywords) can allow you to quickly see what might be contributing to higher or lower CTRs.
  • Layering Googlebot visit data (from your log files) with traffic data (from your analytics integration) can help you better understand why your URLs are seeing higher or lower search traffic.
  • Layering crawl data with log file data allows you to see how many total pages your website has, and which of them Google is/isn’t visiting.

And with Botify Intelligence and Activation to boot, you’ll be able to make even better use of all that data, surfacing high-impact opportunities faster and automating actions to save you resources.

Diving deeper: What else does Botify have to offer?

Now that we’ve covered the main sections of the Botify platform, we think it’s worthwhile to dig a little deeper and talk about some of the features within the platform that make Botify unique.

Starting further back than keywords

One of the things that makes Botify stand out is that we take a URL-first approach to your SEO data, rather than a keyword-first approach. We start by giving you a complete picture of every URL on your website (not just desktop, but encompassing things like AMP versions, alternate language versions, mobile versions, etc.) and then layer onto that everything else you’d want to know about each individual URL, giving you a complete picture.

This makes it easy to identify what could be done at each stage of the search process to increase your SEO performance. And, in January 2020, we’re taking this concept even further by launching technologies like ActionBoard which draw from this data and surface the highest-impact tasks for you.

Data layering & segmentation

Data layering is key to uncovering the “why” behind your data. For example, when your top-performing URLs (a.k.a. pages that get the most organic search impressions and traffic and rank for the most queries), top strategic URLs (those that make your business the most money or support strategic initiatives), or other critical URLs aren’t ranking, it’s time to investigate.

To do so, you’ll need to layer your log file analysis data on top of a given URL to see if and how Googlebot is crawling it. If the page isn’t getting crawled, you can layer the data from your Botify crawl (in Botify Analytics) to see if there’s a correlation between low crawl rate and low content uniqueness (for example).

You can also pair log file analysis data with traffic data, rank data with crawl data, and the list goes on.

Additionally, you can use segmentation to isolate parts of your site for analysis. For example, you can look at pages by author, pages by topic, or all pages on the blog. Looking at specific sections of your site in isolation allows you to identify issues that might be affecting only those pages and to make adjustments accordingly.

Using segments and data layering opens up opportunities to see correlations you would have had difficulty seeing otherwise.

Custom reporting that shows your value & gets buy-in

With so many tools and resources available, it’s not uncommon for SEOs to spend much of their time exporting data and translating it into reports. But when your data lives in one place, everything gets a whole lot better!

With Botify, there’s no longer a need to export your data. You can pull data from Botify Analytics, Botify Keywords, and Botify Log Analyzer into our Data Station, where you can view your report directly or download it if needed. We also have APIs that tie into Google Sheets, Google Data Studio, or your CMS, and we have a Botify Chrome Extension. Plus, if your business uses Looker as its business intelligence dashboard, we have good news. We recently partnered with Looker so you can view your SEO data alongside key business metrics through our custom Botify Looker Block®.

JavaScript rendering, structured data analysis, and so much more!

Because we build Botify directly in response to the problems our enterprise SEO customers are facing, Botify also offers solutions such as:

  • Render JavaScript at scale with our JavaScript Crawler. Track which resources are executed, and collect valuable user-centric performance metrics (time to first paint, time to interactive, and other performance metrics) to ensure your JS content is rendered and seen by search engines. You can render up to 100 URLs per second and conduct a 1 million page analysis in just three hours, instead of waiting days or weeks!
  • Automatically extract 15+ different types of structured data without needing to set up custom extracts to pull information.
  • Evaluate content on your website for quality, parity, and more the same way a search engine would, and eliminate template content from your analysis.
  • Get direct support through our Customer Success Managers, who help our customers make the most of Botify, and our Professional Services team, who are available to answer technical questions and support custom projects.

What our customers love

Our customers have websites with millions (or billions) of pages, which makes their SEO needs rather unique. Due to our fast-changing industry and appetite for satisfying our clients’ needs, we’re always releasing new updates. Whether this means developing backend improvements, like crawl speed or JavaScript rendering capabilities, or UI/UX improvements, we typically introduce new functionalities every 3-4 weeks. Take a look at some of the product updates we’re launching in early 2020 — from urgency alerts to instantaneous indexing!

Our customers can tell that we’ve designed everything for their needs (for scale, speed, granularity of data, security, API features, and more), and we’re excited to bring them the best solutions for their unique challenges.

If you’re curious to know more about how our customers find value in Botify, you can take a look at our customer reviews on G2!

So, what are you waiting for?

Using Botify will help you identify where that potential lies. Expect to be able to save a lot of time and energy uncovering opportunities you may not have seen previously — and fast!

Get more articles like this in your inbox monthly!