Keyword Tracking for Enterprise SEO: Scale, Segmentation, & Real Searcher Data

17th September 2019The Botify Team

Rank position is one of the most well-known and commonly-used KPIs in SEO, but it can pose some difficulties for large organizations with huge websites. In this article, we’re taking a deep dive into keyword tracking for enterprise SEO: how to scale it, manage it, and get to the bottom of why your rankings are what they are.

But first, a refresher.

If you’re a little rusty on the details of keyword tracking, aren’t an SEO practitioner, or are new to the concept of rank tracking, we encourage you to read the next three sections on “What is keyword tracking?”, “Why do you need to track keywords for SEO?”, and “What factors impact keyword ranking?”

If you’re already familiar with these topics, feel free to skip down to the section titled “What makes keyword tracking difficult for enterprises?”

Rather watch than read? We get it! Head on over to our webinar recording where our very own Chase Lyne will walk you through enterprise keyword tracking with Botify Keywords.

What is keyword tracking?

Keyword tracking, also known as “rank tracking,” is the process of identifying what position your URLs occupy on the search results pages for certain keywords.

Most companies, at least the ones that have an SEO program, use software to keep track of their rankings. A popular free option for rank tracking is Google’s own report in Google Search Console called “Performance.”

It looks like this:

Anyone who manages a website can set up a free Google Search Console account. Learn how by reading through Google’s documentation. Google also has something called a “Search Console API,” which we’ll talk about in more detail later.

Why do you need to track keywords for SEO?

Keyword rank tracking is one of the most popular SEO KPIs for a reason: it helps you understand how visible your web pages are in the organic section of Google (and other search engines!) search results. While other SEO KPIs like organic traffic are important, rank tracking helps you understand how you got that traffic.

When choosing a rank tracking solution, make sure you’ll have the ability to not only see what position you’re ranking in, but which of your URLs rank for each keyword. Pictured below is Botify’s rank tracking solution, Botify Keywords, which shows you the URL, impressions, clicks, CTR, and average position of all your keywords.

Why are my keyword rankings fluctuating?

The reason you even need to track keywords is that rank position changes often. You may be ranking in position 4 one day and in position 7 the next. These fluctuations in rank position can happen for a number of different reasons, including:

• Algorithmic updates: Google makes 1-2 updates to its algorithm every day, as well as a few major updates every year. When they adjust the program that’s responsible for ordering search results, that can affect your rank position. Google makes these changes to improve the relevance of search results, so if your rank position declines for a certain keyword, consider how to make your page a more relevant answer to the query.
• Website changes: Any change to the website has the ability to impact how it performs in search results, so make sure you understand who has access to make changes to your website, and what they plan on changing. Make sure you discuss the SEO implications before embarking on any major change.
• Increased competition: Your rankings can fluctuate even if Google’s algorithm and your website remain the same, and that’s because of your competition. You may lose rank position to another website because they published a more relevant answer to the query.

Additionally, even if you maintain the same rank position, your traffic could still fluctuate for reasons such as:

• Demand fluctuation: Even if all your keyword rankings remained the same, your traffic could still fluctuate due to changes in demand. For example, your retail website could be ranking #3 for “Christmas decorations,” which brings in 100K monthly visitors during peak Christmas shopping season, but only 5K monthly visitors in the off season.
• SERP layout changes: Google often experiments with changes to the SERP layout. Common changes that can affect your organic traffic even when your position remains steady include featured snippets and PPC ads, which can both push the traditional organic results lower down on the page, attracting fewer clicks. Because they answer the question on the SERP itself, featured snippets can also lead to “no click searches.”

What makes keyword tracking difficult for enterprises?

Rank tracking may not pose too many difficulties for a small-to-medium sized business, but it can quickly turn into a headache for enterprise organizations. Why? Because enterprises often have larger websites, sometimes multiple websites, and usually more complex websites (e.g. multiple country versions). With more pages and complexity, it becomes even more important to have a scalable, organized rank tracking solution.

Let’s take a look at some specifics — exactly what kinds of issues do enterprise organizations run into with rank tracking?

1. Rank tracking is difficult to scale

Many enterprise websites are huge. Think about an e-commerce site with 8 million indexed URLs or a news/media site that publishes 100 new articles every day. When you consider the fact that a single URL can rank for multiple keywords (sometimes hundreds!), this can quickly get out of hand.

Such a high volume of potential keywords causes three major problems:

• Cost: Many rank tracking solutions are pay per keyword, forcing enterprises either to track everything at exorbitantly high costs, or forego tracking some keywords to save on costs. Even if you use the free Google Search Console performance report, it only gives you 1,000 rows of data, which is why we recommend the API — more on this a little later.
• Missed keywords: Most rank tracking solutions work by having you select which keywords you want to track. The problem with this approach is that it’s virtually impossible to predict all the different queries your pages will show up for, so you’re almost guaranteed to miss out on tracking certain keywords.
• Disorganization: On sites this big, even if you are able to monitor every keyword that’s important to you, it’s difficult to keep track of everything. Imagine wading through millions of lines of keyword data and trying to glean something meaningful!

2. Rank tracking is difficult to maintain

Because many rank tracking solutions require you to pick which keywords you want to track, you won’t be tracking any new keywords your site ends up ranking for, unless you add them manually over time (and we know that “manually” is a very ugly word in the enterprise world!).

Rank tracking typically comes as the result of keyword research, a process in which you determine what phrases your audience is using to search for the information or products you offer. This process can be long and cumbersome, especially for enterprises that want visibility for millions of different keywords. Because of this, keyword research is usually treated as a “set it and forget it” project, or at best, something that’s done quarterly or annually.

If you only perform keyword research once, or very infrequently, you’ll miss out on tracking new keywords that your site is gaining impressions and clicks for.

3. Rank tracking doesn’t give me all the details I want to know

Enterprise organizations often want more granularity than most rank tracking solutions are able to give them. For example, they want to be able to answer questions such as:

• How are we ranking on mobile vs. desktop?
• What keywords are we ranking for by country?
• Which of our keywords are trending up or down?
• What keywords are we missing the most clicks for?

Many rank tracking solutions will tell you what position you’re ranking in for your selected keywords, but can’t give you many of the additional details like how many clicks you’re getting for those rankings. This is because most rank tracking solutions scrape Google search result pages to calculate rank position, rather than use real searcher / real website data. You can learn more about that in our article How To Check What Keywords Your Site Is Ranking For.

4. Rank tracking is missing a lot of context

Similar to the problem of lacking details is the problem of missing context. Enterprise organizations need to be able to know not only what positions they’re ranking in, but why they’re ranking there. Without that context, you’ll have little to go off of when it comes to taking action.

What would make keyword tracking easier for enterprises?

Ideally, enterprise organizations need a keyword tracking solution that allows them to:

• Track an unlimited number of keywords in meaningful segments
• Automatically track new keywords rather than manually adding them
• Understand keywords by device, location, and trends
• Get the context to know why something is ranking the way it is

Botify wanted to deliver exactly that, which is why we launched Botify Keywords. Botify Keywords pulls your keyword performance data from the Google Search Console API and pairs it with the rest of your data in the Botify platform, such as crawl data, log file data, and analytics data.

Let’s unpack that a bit.

What is the Google Search Console API?

The Google Search Console API allows us to pull all your website’s keyword data into Botify, and “all” really means all. In Google Search Console itself, you only get access to 1,000 rows of keyword data. If you use the API, on the other hand, you get all of it.

To learn more about Google Search Console’s keyword capabilities, check out “The Ultimate Guide to GSC Keyword Data.”

What is the benefit of pairing keyword data with my other SEO data?

The benefit of using Botify Keywords to view your Google Search Console keyword data is that you get to layer it with your other SEO data for added context.

At Botify, we like to say that search is a process. As a process, it has a beginning and an end.

• Step 1: Search engines must crawl and render your pages
• Step 2: Crawled pages are added to the index
• Step 3: Indexed pages are ordered by relevance for each query
• Step 4: Ranking pages can be clicked on by searchers
• Step 5: Website visitors can become customers

Most rank tracking solutions only show you Step 3 — where your indexed pages show up for different queries. But to get to that point, your page first needs to be crawled, rendered, and indexed. Without seeing your pages as Google sees them, you may never understand why your pages are ranking in the positions that they are.

Because search is a process, technical issues can absolutely impact keyword performance. For example, take a look at all these technical issues that can cause poor ranking performance and notice that none of them have to do with on-page optimization like content and keywords:

• Page isn’t ranking because Google hasn’t crawled it
• Page isn’t ranking because Google hasn’t crawled it recently
• Page isn’t ranking because we’re not linking to it anywhere on the site
• Page isn’t ranking because it canonicalizes to a different URL
• Page isn’t ranking because it has a noindex tag on it
• Page isn’t ranking because it doesn’t have a 200 status code

…and the list goes on. That’s why it’s critical to be able to pair your keyword data with your technical data.

You may also like our Botify Keywords announcement post where we unpack key features and use cases.

How can I use Botify Keywords in my daily workflow?

It’s one thing to arm yourself with all the data. It’s quite another to incorporate that data into meaningful workflows you can use in your day-to-day.

That’s why we’ve put together some of the top use cases for Botify Keywords.

Identify keyword wins and losses for key segments

Do you know which of your keywords increased or decreased in position, clicks, or impressions? In Botify Keywords, you can set any date range and see whether these important keyword KPIs improved or got worse over that period.

This is great for everyday monitoring, but it can be especially helpful when you’re going through a site migration — just set the date range to cover the time before and immediately following the transition, and see what changes result.

We also recommend looking at your keyword data through the lens of segments. Segments are meaningful groups of your URLs that help you keep better track of your data. For example, you can segment by pagetype, template, content type, product type, and categories. You can even create a segment for your most important pages, that way, your most critical keyword information doesn’t get lost.

Here’s what segmentation looks like in pie chart form:

Without segmentation, it can feel like an uphill battle identifying keyword performance trends and causes in organic search. It’s like searching for a needle in a haystack.

With segments though, you can identify great insights like how your rank position is impacting the number of clicks you get across key segments.

Or how your CTR changes by device across different segments.

Identify “striking distance” keywords

Another key use case of Botify Keywords is identifying “striking distance” keywords — in other words, keywords that could result in substantial traffic wins for your website with just a little love.

In the picture below, you can see that we’ve created a filter that is going to show us all keywords ranking between position 6 and 9 — the bottom of page 1. Keywords in this position are close to positions that tend to get a lot of the clicks, but far enough away that they’re missing most of them.

We’ve also set the filter to show us only keywords that we’re getting more than 1,000 impressions for. This ensures that we’re only looking at the highest-impact opportunities (rather than keywords that would only bring us in a handful of new impressions per month).

Finally, you need to set a CTR threshold. For us 3% made sense (we want to ensure that we’re only going to be working on the keywords that actually need improvement) but keep in mind that each site is different. You can select whichever CTR makes sense for your unique site.

Once you apply those filters and have your list of “striking distance” keywords, you can start to layer on crawl data to help answer questions such as:

• How do the HTML tags stack up against other URLs?
• Is the content we provide on these URLs adequate?
• Are we linking to these URLs appropriately?
• What’s the experience like once a searcher lands on this page?

Like the idea of identifying keywords you’re close to ranking for? You may also like our article on Keyword Research for SEO: How to Create a Killer Keyword Strategy.

Identify how you’re ranking across different locations

Similar to segmentation, identifying the locations you rank in is important for global brands. Botify countries data can help pinpoint any regional cannibalization that is occurring.

For example, the data could reveal that your Canadian visitors are being shown U.S. URLs. Knowing where your searchers are located is a great way to identify whether they’re being served the appropriate content.

Identify whether published pages have been crawled

For most enterprise websites, particularly publishers, it’s incredibly important to ensure that recently published content gets crawled and indexed quickly so it can start attracting traffic.

In the example below, you can see now only when Google has crawled your recently published pages…

…you can also see how that’s translating into impressions and visits.

The use cases definitely extend beyond publishers as well. If you’re launching a new section of your website, for example, this would be a great way to keep track of how quickly Google is finding and indexing those new pages.

Find how internal linking might be impacting your keyword rankings

Another interesting insight you can glean by layering your keyword data with your other SEO metrics is how factors like internal linking might be impacting your rank position.

In Botify, you can visit the “Impact of Inlinks on Ranking” report to see how many rankings you have by the total number of internal links. In the example below, the URLs that are contributing to the site’s performance the most are those that have more than 100 links to them. The chart pictured is showing clicks on the y axis, but you could also change that to rankings or impressions as well.

Sometimes it can be difficult to articulate exactly why spending time on a particular SEO initiative is a good idea, but reports like this make it much easier. Instead of saying, “We should add more internal links because it’s SEO best practice” you could say “We should add more internal links because pages with more internal links rank better and capture more clicks, on average.”

Take your rank tracking to the next level

Most keyword tracking software is just that — keyword tracking. But as you can see, you’re missing critical context if you’re not:

• Pairing your keyword data with your other SEO metrics
• Analyzing keyword data by site segment
• Using real searcher data rather than scraped ranking data
• Tracking all of your keywords, as you rank for them

If you want to see Botify Keywords for yourself, book a demo with us! We’ll walk you through the platform so you can get a feel for how our keyword tracking solution could work for you. Have questions before you commit to a demo? No sweat! You can always reach us on Twitter — just tag your questions @Botify and we’ll get back to you shortly.