Export Exactly What You Need – and Do It Easily

15th May 2014Annabelle

Update: this blog post shows a previous version of the Botify interface. See the current version of the URL Explorer in the Q&As or in this video.

The URL Explorer was named modestly. It is so much more than just an explorer. It could have been named URL Metadata Miner, for instance.

What do we mean when we say that you can easily dig into your data? Here is an illustrated answer.

Dig into your data. But how?

With the URL Explorer, you can explore urls using a multifaceted approach, and you get exactly what you need about those urls. It’s a two-step approach :

• select the list of pages you need, according to a combination of criteria. There are 70 metrics to choose from, starting with criteria related to the url itself, page tag content, links to or from the page, depth in website structure, http status code, etc.
• select the information you want about those pages (from those same 70 metrics – not necessarily the ones you chose to filter on)

This defines how to fill your results table (i.e., the data you can export): the first step selects the lines and the second step selects the columns.

Easy to reach, with pre-filled data

One great feature of the URL Explorer is that you don’t have to enter all this from scratch (although you can if you want to). The initial work is done for you. You can reach the URL Explorer from anywhere in the report, to see data related to an indicator.

Then, you can refine, add, remove information until you get exactly what you want.

And just click to export.

Where to start?

Wherever. When you see something you want to investigate or locate precisely, click on it. For instance:

• a url list for a given indicator (there is always an ‚Äòexplore all urls’ button on the right)
• a Suggested Pattern for a problem (simply click on the pattern)
• part of a graph. For instance, below, the yellow area (urls with a rel=canonical tag that points to another page):

From a graph, a panel will open up on the right, with details on what you clicked on (a sample, in most cases). As all lists, it includes an ‚Äòexplore all urls’ button.

Enter through an indicator

A click on ‘Explore all urls’ bring you to the URL Explorer, pre-filled with the same data. All of it, this time:

This is only the beginning.

Once in the URL Explorer, you can adjust the selection to your the urls you want to focus on, and get all the information you need about these urls.

Let’s take another example. Imagine that you see this in the ‚Äòinlinks’ section:

And you go to the URL explorer to see these 18K urls that are linked only once. You not only get the url list, but also their http status code and depth distribution, in separate tabs:

This can help narrow down your selection, by providing hints on filters to add. Only deep urls, for instance.

Divide the problem, get actionable data

Let’s continue with our example with urls linked only once, and take a look at the http status code distribution tab:

Hover your mouse over the blue area, and you’ll learn that among these 18K urls with a single incoming link, 739 (4%) are redirected.

Of course, you won’t have the same approach for redirected and non-redirected urls:

• good urls (200 http status code): you will want to add links to these pages, once you have checked that they are quality pages that are good targets for organic traffic. Suggested Patterns can provide additional insight to locate these pages on the website.
• redirected urls (301 http status code): you will want to know where they are linked from, and where they are redirected to, to make sure they link directly to the correct url. And you’ll probably want to do this, not only for redirected urls that are linked only once, but also for those that are linked more than once.

Zoom in

Let’s work on the 301s. As a next step, you can zoom in on these redirected pages, by adding a filter (http status code = 301). To do so, click on the green ‚Äò&’ to add a filter, select http code in the drop down list, and enter ‚Äò301′ in the value field.

And you can check how many more there are, if you include pages with more than one incoming link: change the filter for incoming links to ‚Äògreater than one’. As it happens, in this example, there are few : only 49.

So it makes sense to start working on the 700+ redirected urls with a single incoming link: it’s probably easier to change (in all likelihood, a small change in one template), and has a higher impact – once you have validated that the pages they redirect to are important pages.

Get what you need

Now that you have narrowed down the set of urls you want to work on, you need to get additional information about these pages:

‚Üí add a column for ‚Äònumber of incoming links’ (all of them this time, not the ‚Äòfollow’ type only, as you’ll want to remove this url completely from the website),

‚Üí add a column for ‚Äòredirects to’.
To do so, click in the empty area in the ‚ÄòFields to display’ and select the new column in the drop down list, or start typing to narrow down the choice.

Then click on ‚ÄòApply’ and you’ll get this:

All listed urls are cliquable, if you want to see indicators on any of them. So you’ll find out where the redirected urls are linked from.

This was a simple example, but you can go very far in the selection of urls, as you can combine filters with ‚Äò&’ and ‚Äòor’.

Last but not least: export the result of your investigation

Once you have the table with all the information you need, click on ‚ÄòExport as CSV’. The size limit for exports is currently set to 100 000 lines. But it’s easy to reduce the size of the table with an additional filter. That way you can divide the larger table into subsets that make some sense – instead of just getting the first 100 000. Anyway, this export size limit should be removed soon!

After downloading the flat file, import it in your favorite spreadsheet software and choose a coma-separated format in the import wizard. The first line will be a header.

You’re all set to start removing problems on your website!