Login
Request a Demo
Login
Request a Demo

The Organic Search

Performance Standard

An insights-led content initiative that shares the real organic search performance of enterprise retailers.

What's it all about?

The Organic Search Standard shares the performance of Botify-optimized websites throughout the crawl, index, rank, and visit stages of the organic search funnel.* Botify analyzed the organic search results of 200+ million URLs across 75 of our eCommerce retailers throughout each stage of the organic search funnel. The results present an Organic Search Standard of benchmarking standards that your brand can use to see how you measure up--and how you can develop goals for your own SEO strategies.

Fast Findings

Low URL Count
Medium URL Count
High URL Count
Low URL count: less than 100k site URLs

Of all total URLs

67%

get crawled.

Of all total URLs

24%

are visited.

One crawled URL provides

108

visits.

Of all product detail pages

78%

are crawled.

Category pages account for

21%

of all visits.

Medium URL count: between 100k and 1 million site URLs

Of all total URLs

58%

get crawled.

Of all total URLs

12%

are visited.

One crawled URL provides

18

visits.

Of all product detail pages

60%

are crawled.

Category pages account for

40%

of all visits.

High URL count: more than 1 million site URLs

Of all total URLs

37%

get crawled.

Of all total URLs

2%

are visited.

One crawled URL provides

2

visits.

Of all product detail pages

46%

are crawled.

Category pages account for

40%

of all visits.

Organic search performance is heavily influenced by a site’s URL count, and this report segments performance by low, medium, and high URL counts to provide a relevant view.*

‍
Strict aggregation measures are employed to ensure customer anonymity. These measures include requirements on comparison set size, diversity, and consistency in order to present credible and reliable information that is insulated from concentration risk and cannot be reverse-engineered to identify any specific retailer. Additional data hygiene factors are applied to ensure accurate metric calculation.

Each site included in the analysis set was actively powered by Botify during the analysis period, September through October 2021.The visit attribution window is an industry-standard 30 days, meaning that visits attributed to a URL that was crawled by Google during the analysis period were attributed to that URL and included in the results.

Data footnotes are included throughout the report to provide additional clarity on analysis.

The Organic Search Standard is not directly indicative of the operational performance of Botify or its reported financial metrics.

Journey of a URL

Understanding organic search requires a full-funnel view, from known URL all the way through to a click. From the universe of known URLs, only a select set of URLs earns a shopper visit. This journey of a URL provides a clear view of how well search engines see and rank a site’s URLs, and how likely shoppers are to click or tap a search result.

While URLs fall out at each stage of the funnel, retailers feel it most acutely at the very first stage of the funnel; Google can’t and doesn’t crawl all URLs. This crawl rate is heavily influenced by a site’s URL count, about 1/3rd of URLs are not crawled on low URL count sites,  while the largest sites – those with more than 1 million URLs – on average see more than 1/2 of their URLs overlooked.

Explore the SEO funnel

The most important step of the journey is up to the shopper, however.  Does she click or tap a link that has progressed to a search results page? Low-URL count sites see 1 in 6 known URLs advance all the way through the funnel to earn a click, while just under 1% of a large site’s known URLs are active.

Go to SEO-Funnel

URL Survival Rates

Like the journey of a URL, which shows the cumulative impact of URL attrition, funnel survival rates shine a light on performance at each stage of the journey. Specifically, how many URLs advance from one stage of the funnel to the next?

Crawled URLs are further whittled down based on what gets indexed. This stage proves to be much less sensitive to URL counts and much more dependent on a retailer’s SEO rigor and hygiene. Index rates for all site sizes cluster near 80%.

Ranking may be the most celebrated stage of the journey and the source of more strategic SEO effort than any other  step. Low URL count sites far outperform the large URL count sites, thanks to a number of factors,  most notably content relevance and focus.

The all-important final step is in the hands of the shopper. Of the links offered on a search results page, which ones are clicked or tapped? This is the only step of the journey where the high URL count sites outperform the low, though the difference is very slight.

Getting Your Pages Crawled, Rendered, and Indexed

Check out The Future of Enterprise SEO and An Enterprise SEO Methodology: From Crawling to Conversions.

How to optimize for crawl budget

Content That Ranks for Real Searcher Queries

Check out The Future of Enterprise SEO, An Enterprise SEO Methodology: From Crawling to Conversions, and Crawl, Render, Index.

Learn more

The Active URLs

URLs are considered active when they earn a shopper visit. Of links that are crawled, the share of active URLs ranges from 24% for the low URL count sites, to 2.2% of the high-URL count sites. Active URLs tell only a piece of the results story, though. The more important measure is how many visits those active URLs generate.

Delivering Organic Visits

The ultimate measure of success of SEO goes beyond crawling, indexing, ranking, and even links. The goal - of course - is to drive traffic and, ultimately, conversions. To truly understand the value of a URL, retailers need to know the number of visits each active URL earns on average.

The Value of a Crawl

How important is a single URL? For every crawled URL, retailers see more than 100 visits on low-URL count sites. On the largest URL count sites, a single crawl delivers about 2 incremental  visits. Given the breadth of these large URL sites, the power of SEO’s scale is on display: even at the highest URL volumes, visit volume outpaces the URL count.

Spotlight on: Page Types

The product detail page (PDP) and category page stand alone as the pillars of e-commerce pages. Together, these two pages account for nearly 70% of a shopper’s on-site experience. So, too, these pages play a leading role in attracting shoppers, delivering 46% of all organic traffic (nearly identical to the 47% of traffic driven by the powerhouse homepage). Here’s a look at how these two pillar pages compare.

70%

of all users

on-site experience from PDP

Understand exactly how Search Engines see (crawl) your website 

Identify what high-value pages aren’t receiving any organic traffic 

Uncover the real keywords used by your customers 

46%

of all traffic

nearly identical to the powerhouse homepage

Understand exactly how Search Engines see (crawl) your website 

Identify what high-value pages aren’t receiving any organic traffic 

Uncover the real keywords used by your customers 

Page Shares

Product detail pages are the source of an e-commerce site’s breadth, serving as the atomic unit of a retailer’s catalog. To wit, the PDP accounts for the overwhelming majority of crawled URLs, outpacing category pages by a 4:1 ratio across all segments.

Through the SEO funnel, the PDP majority remains, though it ticks down, as category pages take a bit more URL share at each stage. When it comes to visit volume, though, category pages are the site’s darlings, flipping the script to far out-visit PDPs across all segments. Each page serves an important purpose for the retailer: PDPs satisfy the long-tail needs and tend to be the highest converting pages from organic search, while category pages generate high-volume demand.

URL Survival Rates By Page Type

Comparing the funnel performance of the top two page types highlights the dominance of product detail pages. At each stage of the journey, PDPs outperform category pages. Early in the funnel, PDPs are well-crawled, providing long-tail benefit for retailers. Even index rates are higher for product detail pages, though low URL count sites earn the highest index rates across segments. Later in the funnel, the laser-focus of a product detail page provides a bastion of relevance, delivering superior ranking rates from Google and healthy visit rates from shoppers.

PDP crawl and index rates again outpace category pages for the medium and high URL count segments. Later in the funnel, performance on medium and high URL count sites differs from their lower URL counterparts. For medium and large sites, category pages outpace PDPs, earning higher rank rates and stronger visit rates.

Visit Density

Category pages are magnetic, attracting shoppers and delivering far stronger visit density than their peer in popularity, the PDP. A crawled category page earns far more visits per page than a crawled product detail page. For low-URL count sites, crawled category pages see 9 times more visits than crawled PDPs. For high-URL count sites, the multiplier is 12x. The message is clear: leave no category page behind. Category pages are the workhorses for generating demand while product detail pages are a volume play, capturing long-tail demand.

Methodology

Botify tracks, monitors, and optimizes the performance of more than 500 customers across the globe. The Organic Search Standard analyzes Ecommerce site performance at each stage of the organic search funnel, starting with a site’s known URLs and ending with those that trigger shopper visits.

About Category pages

Category pages are categorized to include similar pages, including brand and collection pages.

Understand exactly how Search Engines see (crawl) your website 

Identify what high-value pages aren’t receiving any organic traffic 

Uncover the real keywords used by your customers 

Analysis powered by Botify

Each site included in the analysis set was actively powered by Botify during the analysis period, September through October 2021.

These insights are drawn from Ecommerce activity during Q3 and Q4 2021, representing 200 million URLs, more than 500 million site visits across 75 leading retailers.

Understand exactly how Search Engines see (crawl) your website 

Identify what high-value pages aren’t receiving any organic traffic 

Uncover the real keywords used by your customers 

URL Count

Sites are categorized by URL count. All metrics presented represent the average performance of each URL count segment.

Under 100,000k site URLs: Low URL count
Between 100,000 and 1 million URLs: Medium URL count
More than 1 million URLs: High URL count

Understand exactly how Search Engines see (crawl) your website 

Identify what high-value pages aren’t receiving any organic traffic 

Uncover the real keywords used by your customers