Crawl & Render Budget E-Commerce SEO Enterprise SEO Performance & Speed Ranking Factors Uncategorized

How Your Brand Can Set the Standard in Organic Search Performance

For the first time in a decade, e-Commerce is in a drought. 

Over the last three quarters, e-commerce web traffic has fallen and growth has evaporated across all verticals.

Compounding this traffic issue, retailers are increasingly reliant on paid ads to acquire new shoppers, and the cost of that traffic is increasing. As a result, paid traffic now accounts for a larger share of visits than ever before with paid search and social combined driving 25% of all traffic, up from 20%  last year. 

So, retailers are in a sense spending more to attract fewer shoppers. 

Gaining back a larger share of that reduced amount of traffic now has become an even more critical e-commerce investment for all organizations. 

To overcome the drought and costly external traffic sources, leading retailers must look inward and unlock their own untapped organic search potential. 

An achievable organic search standard 

The Organic Search Standard (OSS) is a newly developed benchmark of organic search performance in e-commerce, drawing on the crawl activity of more than 200 million URLs and generating more than 500 million shopping visits across 75 global retailers—all powered by the Botify platform. 

These insights can be used to give e-commerce brands a realistic and relevant set of goals to strive for and emphasize multiple opportunities for retailers to attract more shoppers without relying on paid traffic sources.

 Hear More about the OSS

The inaugural data compiled in the OSS shines a light on three insight areas that help retailers benchmark performance and identify growth opportunities in their organic search program:

  • Journey of a URL: where attrition impacts organic search performance 
  • The value of a crawl: how many visits is a crawl worth?
  • Know your pages: organic search performance by page type

Journey of a URL

The first step to unlock your brand’s untapped organic search potential and develop a benchmark is to identify the journey of each of your URLs.

The journey of a URL tells the step-by-step story of a website’s organic search performance. It begins with the set of URLs you wish Google to crawl and progresses all the way to—hopefully—a shopper’s click. 

But the reality is that Google can’t and doesn’t crawl all URLs, and for the ones that it does, there’s going to be attrition along the way, otherwise known as a crawl budget.

The below graphic depicts the journey that a URL goes through in an attempt to reach a consumer–and the stages that attrition happens along the way.

Optimizing organic search must start with retaining as many URLs at the start of the journey as possible, making the most of the crawl budget.

URL attrition is most pronounced at the very start of the journey, particularly for sites with larger URL counts. Low URL count sites fare better at an average crawl rate of 67%; but for sites with more than 1 million URLs, that rate dropped to just 38%. 

Fast forwarding to the end of the journey, our research shows that on low URL-count sites, 1 in 6 known URLs advance all the way through the funnel to earn a click. For high URL-count sites, just under 1 in 100 make it that far. 

Understanding how many of your URLs make it to visited or clicked can help you optimize your other pages to ensure they aren’t ignored by search engines. Our research based on URL counts can help you benchmark where you should be compared to other similar-size retailers–and where you should strive to be after you determine how many visits a crawl is worth in your organization and ensure the performance of your specific page types.


How can Botify help you reach the OSS? 

The Value of a Crawl

In search, optimization happens throughout the URL journey, but some of the juiciest opportunities are at the very beginning during the crawl phase. 

The ultimate measure of success of SEO goes beyond crawling, indexing, ranking, and even links. The goal – of course – is to drive traffic and, ultimately, conversions. To truly understand the value of a URL, retailers need to know the number of visits each active URL earns on average.

The effects of a single additional crawl trickle down to the conversion stage, revealing the value of a crawl. The chart below depicts from our research just how many visits URLs receive when they are crawled versus active based on the number of URLs across a site. URLs are considered active when they earn a shopper visit.

For sites of all sizes, improving crawl rate is essential. For every crawled URL, retailers see more than 100 visits on low-URL count sites. On the largest URL count sites, a single crawl delivers about 2 incremental visits. 

Active URLs tell only a piece of the results story, though. The more important measure is how many visits those active URLs generate.

Given the breadth of these large URL-count sites, the power of SEO’s scale is on display: even at the highest URL volumes, visit volume outpaces the crawl volume. So, for every 1,000 additional crawled URLs, large URL-count sites generate 1,643 additional visits.

Know Your Page Types

Understanding how different page types perform is another method retailers must use to ensure they reach the OSS shines. 

Our research spotlights the most critical e-commerce pages to our customers—the product detail page (PDP) and Category Pages—and how their performances differ throughout the SEO funnel. 

The PDP and category page stand alone as the pillars of e-commerce websites. Together, these two page types account for more than 70% of a shopper’s on-site experience, according to YOTTAA’s Site Speed Standard. They also play a leading role in attracting shoppers, delivering 46% of all organic traffic (nearly equal to the 47% of traffic driven by the powerhouse home page). 

The PDP is the highest volume page type, of course. As every product has its page, retailers’ URL counts are driven primarily by the number of PDPs.  PDPs account for between 70% and 78% of all crawled URLs across sites of all sizes.

So the PDPs deliver the majority of “active” URLs—URLs that make it all the way from crawl to click—ranging from 53% for high URL-count sites, to 81% for low URL-count sites.

Where it matters most, though, it is the category page that really stands out. Category pages out-visit PDPs, contributing triple the number of visits.

Each page serves an important purpose for the retailer: PDPs satisfy the long-tail needs and tend to be the highest converting pages from organic search while category pages generate high-volume demand.

At a time when retailers need a path to growth and a way out of the recent traffic drought, organic search presents a compelling opportunity to source visits without the ever-increasing cost of paid traffic. 

The Organic Search Standard provides a barometer for retailers to use in assessing their website’s performance relative to e-commerce sites of a similar size. With a comprehensive view of the organic search journey, it will be easier for digital marketers to identify high-yield optimization opportunities. 

Learn more about how Botify can help you reach the OSS. 

Sep 9, 2019 - 10 mins

Keyword Tracking For Enterprise SEO: Scale, Segmentation, & Real Searcher Data

Crawl & Render Budget E-Commerce SEO Enterprise SEO Performance & Speed Ranking Factors Uncategorized
Mar 3, 2017 - 2 mins

Top 5 E-Commerce SEO KPIs

Crawl & Render Budget E-Commerce SEO Enterprise SEO Performance & Speed Ranking Factors Uncategorized
Mar 3, 2022 - 8 mins

What Is Faceted Navigation? The Main Factors to Consider for SEO

Crawl & Render Budget E-Commerce SEO Enterprise SEO Performance & Speed Ranking Factors Uncategorized