The Future of Search

Search, as we know, has been around for a long time. While there’s probably an interesting academic discussion happening somewhere about the true definition of “search,” and technically how old the idea of search is, for simplicity’s sake, let’s define it as the modern, Google-driven era we’ve all grown accustomed to. 

For people of a certain age (like myself), this era coincides with their professional development. I hunted for things online in high school through manually curated Yahoo directories and hit-or-miss search engines like AltaVista. By the time I finished college in 2004, I had a Gmail account and expected to find whatever I needed with the correct search query. Modern search life was here, and it came with certain expectations and assumptions. Those included:

• Online you can find anything.
• The way you search is by sitting in front of a computer and typing.
• Optimization is about reading Google’s mind.

People responsible for daily SEO work know these assumptions have weathered various technical and policy changes over the twenty years and that they are more nuanced than they appear. But despite that, this is still how most people (searchers and people looking to attract them) think about search. That mentality defines how we prepare for search, adapt to its requirements, and invest in its future. 

For the most part, that’s been… fine? Search has certainly changed on the margins, with more paid results continuing to creep into results pages. Companies like Botify have found that even if you assume the basic 2004-era logic of search remains intact, there still may be a smarter, more efficient way to optimize for it. But on the whole, things aren’t fundamentally different than before, and that’s a big reason why organic search strategy is often reactive, risk-averse, and primarily focused on small-bore tactics and vanity metrics like Share of Voice. You may need to plug in 2021-level tactics (Core Web Vitals are coming!!!) to be successful. Still, you can fill them into the same fundamental 2011 strategy of guessing what Google is thinking and worrying about SERP positions without anyone questioning why one of your essential digital strategies is philosophically identical to what it was a decade ago. 

But here’s the thing: many of those “fundamentals” are actually at a clear tipping point for the first time in decades. “Search 1.0” (or maybe “2.0” if you consider those Yahoo directories from high school to be 1.0) is straining at the seams. These fundamental changes are like climate change; they aren’t coming. They’re here. Now it’s just a matter of us experiencing the impact more every day until the things we took for granted for long don’t work anymore. 

So let’s talk about those fundamental changes one at a time.

Fundamental Change Number One: The Web Is Too Big

From 1998 to 2008, the web grew to roughly 1 trillion pages. Over the next ten years, that total grew to over 130 trillion pages. To say that “the internet” or “the web” has grown more quickly than anyone could have anticipated is an understatement. As a result, we’ve started to run into limits that run counter to how many of us expect the online world to work. One of those limits is the ability of search engines to index “everything” on the web — something we’ve assumed is doable by the spiders and search bots that crawl the web every day. In the old world, the main concern was just how discoverable an engine would make content, or maybe whether a technical issue on your end could prevent content from being indexed. Those things are still issues, but they’re increasingly secondarily to simply getting legitimate content indexed at all, even if that runs against the “all-knowing” view of search engines many of have in our heads. 

There are several reasons why the web has outgrown our ability to reliably crawl it (more on that in the next section), but don’t underestimate the simplest problem of all. There’s simply too much ground for a centralized search engine to cover on its own. We’re already living in a world where search engines have to decide what’s worth crawling and what isn’t, and that’s despite major improvements to bots and search tech over the years. Engines have gotten better, but not as fast as the internet has grown. With trends like user-generated content and even machine-generated content that have been integrated into the culture of the web, it doesn’t look like exponential growth is about to slow down. And that means content will increasingly run the risk of being entirely excluded from organic search, something that felt impossible just a few years ago. 

Fundamental Change Number Two: The Web Isn’t Really the Web Anymore

The search bot/spider/crawling paradigm we’ve built an entire digital economy around will go down as one of the truly brilliant innovations of the information era. That being said, there’s no getting around the fact that it was originally designed to parse static HTML content, and less and less of the web consists of that. Instead, we live in a world of dynamic content powered largely by JavaScript code that must be executed first. In fact, even some of our static content is managed by JS-based frameworks and applications, so the problem often extends beyond interactive experiences and into run-of-the-mill content. While we’ve moved beyond Flash-style browser plugins and run-time environments that completely isolate content from search, the modern web’s dependency on constantly running code puts up a huge barrier to search engines. That’s because while search engines can read static HTML a lot faster than people, they can’t run JavaScript code much faster than the ordinary browser you’re looking at right now. When that barrier is put up across trillions of URLs, you start to legitimately threaten the ability of engines to index huge amounts of content with any regularity. They don’t have the bandwidth.

Even more jarring, the problem isn’t getting better — the responsibility for solving it is just getting passed around to different people in an attempt to allow an antiquated model to keep working the way we expect it to. Pre-rendering is a powerful tool for mitigating the cost of JavaScript to users, bots, or both (depending on how you do it). Still, it doesn’t eliminate the cost of rendering static content for search engines as much as passing it on to you. Then again, if you don’t have an easy enough path for search bots, you’re the one who runs the risk of your content not being indexed and paying whatever the business cost of that ends up being. These concerns affect the biggest sites first but are trickling down to more properties around the web as both those sites, and the web as a whole continues to grow in size and complexity. 

Something has to give, and whatever it is will give sooner than we think. Is crawling itself on the way out? Will we submit more content directly through systems like Bing’s API? Will processes like that be automated, outsourced, or something else? All of that is yet to shake out in an industry conditioned for years to simply layout their content properly and get out of the way.

Fundamental Change Number Three: Competition is Coming From Every Direction

Search — just the raw ability to find stuff — is a business. In the 21st century, it’s a massive business with battlefields that go far beyond the 17-inch CRT I used to make my first Google search back in high school. Whether it’s new search-enabled (or even search-driven) hardware like speakers, phones, tablets, TVs, kitchen appliances, etc., or anything else you can imagine, we’ve obviously moved beyond just the desktop. Is Google Maps a search platform? What about the built-in iPhone search function? Alexa? Yes, the answer to all of this is they’re all controlled by different players with different goals, business models, and corresponding ecosystems. 

Even the relationship between organic and paid search is in flux; organic remains the trusted, essential core of search, but never-ending growth goals have spawned more paid search results. That may not change the value of organic (it may actually increase it), but advertising-driven search has already spawned competitors with entirely different approaches to search. DuckDuckGo doesn’t track. Neeva doesn’t even have ads.

Will one of these models catch on? Even if they only attract enough interest to become relevant to a few key demographics, getting content discovered is about to involve serving many different platforms and services. The days of reading the tea leaves from a single source are ending quickly. 

Fundamental Change Number Four: Organic Search is More Valuable than Ever

In many ways, organic search is under assault. It’s under technical assault as the sheer magnitude and function of the web makes search harder to execute, and it’s under strategic assault as existing providers push paid results as far as they can. New competitors bifurcate the market and create new mouths to feed. 

And yet… all of these things somehow make organic search even more valuable. An influx of ads and personalization make “the truth” even more desirable (and noticeable!). Targeted digital advertising leans on tracking and is under enormous pressure from consumers, platforms, and even lawmakers as customers and society develop stronger opinions about data collection. Underneath all of it, customers want to find what they’re looking for, whether that’s an answer, a product, a person, an idea, or an opportunity. Organic search does exactly that, and that’s why not only is it here to stay (regardless of how it gets executed), it’s only going to continue to grow in importance. Over the last year, more people searched for more things and acted on what they found (often by purchasing something) than ever before, and that’s not a trend that’s going anywhere in a world that just spent the last year doing about everything online. 

The Future of Search Optimization

SEO is, by internet era standards, an ancient term. It references “optimization” for a reason — the work of making your ideas discoverable has always been a matter of refining something that fundamentally works, and almost as importantly, basically works one way. 

Like it or not, that world is rapidly leaving us behind – if it hasn’t already. The web isn’t a thing we visit — it’s something we essentially never leave, and it has the commensurate volume, complexity, and increase the number of channels you’d expect for something like that. Is preparing your brand, your products, and what you have to say for that world optimization? Or is it something much bigger, broader, and less tactical than that? Because at the end of the day, sure, you might be responsible for making sure your organization shows up when someone types something into a web browser and hits “search.” But that’s not really the point. The point is to get discovered, or more realistically, discoverable not through one site or even one technology but across a world of channels, devices, and mediums. 

The web is underneath all of that, and many of the technologies we’ve grown up with will continue to power it, so it’s a good thing so many of us are so familiar with crawling, rendering, and ranking. That stuff isn’t going anywhere, but we’re going to have to start applying it to so many different places and contexts (how do you “rank” on a smart speaker?) that brands are going to have to change some fundamental things of their own if they want to thrive.