The Push Revolution: Empowering findability in Technical SEO

Thomas Grange, Botify co-founder and Chief Innovation Officer

Based on a Keynote presentation at brightonSEO.

SEO is (still) not dead

For as long as I’ve been working in the organic search industry, people have been declaring its demise. But the latest headline-grabbing developments (ChatGPT, Large Language Models, and new search interfaces), rather than sounding the death knell of our industry, signal its disruptive vitality. As we recently remarked, search is now commanding global attention and industry-wide investment on an unprecedented scale.  

In truth, search and SEO are evolving. The question is, how can we help that evolution happen, in the right direction, and at a pace that has the most and widest positive impact for search and those it serves?

Here are my views.

The Pull Paradigm emerges 

Before we can embrace that bright future, we need to fix some unhelpful legacies from the past. 

The biggest legacy holding us back is what I’m calling the Pull Paradigm, with robots issuing outgoing “pulls” to acquire content from websites. This timeline illustrates how this evolved as the norm when websites were a whole lot smaller and simpler. 

The concept of “crawl budget” was not very explicit back then. But during this epoch, developments such as robots.txt, sitemaps, link rels, and other canonicals came about to help search engines better target content updates. Such initiatives were more or less successful in helping crawling and indexing efficiency. The system could just about cope. 

But then, somewhere around 2010 search innovation dried up, and we entered The Drought Years.

The Drought years (2010-2019)

From one perspective this decade was transformative. We had Smartphones; Multi Devices. New Web Frameworks all making their debuts. Websites (notably in e-commerce) became bigger and more complex. JavaScript started transforming user experiences. 

Yet from the search perspective, it was a true drought. The infrastructure failed to innovate to keep pace with change. We’re still dealing with the hangover daily. 

Search engines are no longer able to swiftly crawl, render, and surface every relevant page, resulting in pages being missed or incorrectly prioritized, and bots squandering budget and CPU simply rendering pages. 

Even in websites that use mostly HTML, there are often third-party components such as reviews or faceted engines that remain in JavaScript. These third parties are often hosted elsewhere, and sometimes can be complicated to render, or blocked by cookie management systems, which bots can not accept.

Pull over: it’s time for a change of direction

It’s clear that the existing model is slow, inefficient, and unsustainable. The Pull epoch is over. We need a radical change of direction. 

If Pull involves robots pulling content blindly from websites, Push involves the exact opposite – with engines responding to updates coming from websites. 

A total paradigm shift, which promises a better use of resources, crawl budget, and energy consumption. 

Think about it, there are currently large websites spending millions of dollars and carbon emissions in server usage just to have bots blindly crawling their pages, whether there are any updates or not. 

Push provides a more sustainable alternative, taking the pressure off a system that’s already close to breaking point. 

Pushing towards Push  

There are already signs of a gradual, but uneven, move towards Push. 

• IndexNow involves Push, and is arguably the most significant initiative we’ve seen for years. We just ping the search engine to quickly visit the updated pages.
  
• Google’s Indexing API, launched in 2018, is another Push initiative. Yet, this is currently limited to a certain type of page, with no signs from Google of taking this further.  

• And there’s Bing’s own Content API, launched in 2019. This deserves some serious consideration. 

What Push can achieve  

To see Push at work you only need to consider the Bing Content API. With this model websites can send HTML content directly to the search engine. Like IndexNow on steroids, it shows how efficient Push can be. Here are a few results we’ve witnessed at Botify.

(Source: Botify internal research 2022).

We noticed that on a number of sites, a large proportion of pages generating visits from Google were not being crawled by Bing. Using the Bing Content API, we then pushed all missing web pages, and the results were quite impressive: in a matter of days, the traffic increased by more than 10%.

Join the Push Revolution 

To be clear, neither I nor Botify, are promoting one search engine over another. Despite the current arms race going on between the major players around LLMs, the search ecosystem works best when there is openness, collaboration, and collective problem solving. This is what we endorse, innovating as partners in the ecosystem. 

Systemic problems demand systemic solutions. Everything points to Push as a more efficient and sustainable model that serves website owners, search engines, and the environment alike.  

So let’s push for Push together, to help evolve search innovation for all.