SpeedWorkers: Reliable Dynamic Rendering That Gets Your Important Content Found & Driving Revenue

JavaScript is everywhere on the modern web, but it wasn't always this way.

We recently spoke with Martin Splitt, who explained that the web was originally a document platform composed entirely of HTML.

Over time though, websites evolved from pure information to application through the use of JavaScript. However, JavaScript doesn't always just add interactivity to pages. It can also contain information.

Because JavaScript can contain information, Google and other search engines render it in order to fully understand our pages. However, that doesn't always happen successfully or immediately (see: render budget), and it can also add seconds of load time per page, introducing speed-related crawl budget issues for large sites.

Some organizations have attempted to solve for this issue by implementing server-side or pre-rendering solutions, but those can be incredibly resource-intensive and not always reliable.

That's why we built SpeedWorkers.

• What is SpeedWorkers?
• How does SpeedWorkers work?
• Why optimize for bots?
• Where does SpeedWorkers fit into the Botify suite?
• What does the SpeedWorkers interface look like?
• What results does SpeedWorkers produce for customers?

Rather watch than read? No problem! Request your copy of our SpeedWorkers video explainer here.

What is SpeedWorkers?

SpeedWorkers is a dynamic rendering solution that sends fully-rendered versions of your pages to search engine bots.

💡 Dynamic Rendering sends fully-rendered content to search engines while serving human visitors with normal, client-side rendered content. It's pre-rendering, but just for search engine bots. Learn more about dynamic rendering.

This provides customers with two main benefits:

• Pages are delivered to search engine bots faster, solving speed-related crawl budget issues.
• Pages are delivered to search engine bots fully-rendered, solving "partial indexing" issues (i.e. search engines missing the portions of your pages that require loading JavaScript to see).  

💡 Crawl budget is the number of pages on your site that Googlebot can and is willing to crawl in a given session. Every site's crawl budget is different, and is determined by factors such as how fast your pages load, how many errors your pages serve, the popularity of your pages, and more. Learn more about crawl budget.

Dynamic rendering is a legitimate, Google-approved solution to these problems that does not constitute cloaking, as Martin reassured us during our dynamic rendering webinar.

Our promise with SpeedWorkers is to:

• Deliver pages to bots as quickly as possible
• Cache all your important content, not just a few top pages
• Operate at scale, accommodating even the largest sites
• Keep cached content in sync with live content
• Give customers consistently great performance so they can "set it and forget it"
• Provide quality control measures so you know if and when anything isn't rendering properly
• Be completely compliant with search engine terms & best practices

💡 Need some more background on JavaScript before diving in further? Learn more about how JavaScript works, the difference between client-side and server-side rendered content, and how it all affects SEO in JavaScript 101 for SEOs.

How does SpeedWorkers work?

SpeedWorkers operates at the CDN (content delivery network) level.

💡 A content delivery network (CDN) is a network of distributed servers that can help minimize loading delays by reducing the physical distance between the user requesting the page and the server sending the page's contents.

This allows us to read, render, and cache your CDN content on Botify's servers continually, and at a rate you can customize.

This equips Botify with an up-to-date collection of all your site's content, fully-rendered, without you or your CDN needing to do anything on a regular basis.

Being at the CDN level allows SpeedWorkers to intercept bot requests and deliver high-speed, bot-optimized content from Botify's servers in response.

We went this route with SpeedWorkers for two main reasons. For one, CDNs are extremely close to search engine requests, so we can react very quickly. Second, CDNs are very used to supporting third party applications.  

The first iteration of SpeedWorkers is compatible with two of the most popular CDNs: Cloudflare and Akamai. If you use another CDN but are still interested in SpeedWorkers, just let us know! We're already in the process of adding support for additional CDNs and would love to know what you use so we can prioritize that.

This solution means human users are not affected at all. They will continue to receive the normal, client-side rendered content. Their requests for your content never even go to Botify. They're processed normally by your CDN.

Why optimize for bots?

Since SpeedWorkers, like other dynamic rendering solutions, only pre-renders content for bots, you may be wondering why you wouldn't also take the burden of rendering JavaScript off your human visitors as well.

There are a few reasons for this:

1. Humans benefit from usability, personalization, and interactivity, which require JavaScript, so in most cases you'd want to keep client-side rendering your JavaScript.
2. Pre-rendering content for both human users and bots is often cost prohibitive.
3. Solutions like server-side rendering often require more resources to develop and maintain than many engineering teams have time for. Also, if you use a lot of third-party services on your pages (e.g. review feeds), they won't be rendered with server-side rendering.
4. Manually making performance improvements can improve load speeds for both humans and bots, but those projects are often lengthy, taking up to a year or more to see measurable improvements.

We want to make our website experiences enjoyable for people, but at the same time, people aren't the ones that crawl and index the web. Search engines do. And that's why it's important to optimize for bots.

💡 Because SpeedWorkers focuses on providing fully-rendered content to search engines, it would not improve your Core Web Vitals metrics, which measure real user interactions with your web pages. Learn more about Core Web Vitals.        

Where does SpeedWorkers fit into the Botify suite?

The Botify platform comprises three suites:

• Botify Analytics is where you can identify problems
• Botify Intelligence is where you can discover the highest-impact solutions
• Botify Activation is where you can implement those solutions

SpeedWorkers falls under Botify Activation.

But even though SpeedWorkers sits within the Botify Activation suite, it integrates and interacts with Botify Analytics and Botify Intelligence as well.

For example:

• Since SpeedWorkers intercepts all search engine requests of your pages, all the logs from your dynamically rendered pages can be seen in LogAnalyzer in real time.
• SpeedWorkers data will be blended with EngagementAnalytics data, allowing you to see the amount of money you spent on SpeedWorkers for a given period alongside the revenue you've gained during that same period from organic search.
• Crawl frequency information that we glean from your site's LogAnalyzer data can be used to estimate when you should refresh our cache of your pages. This ensures the freshest version of your pages is being served to search engines as efficiently as possible. In the future, Botify customers will be able to surface this information in ActionBoard as well, so it saves even more time, and consequently, money.

What does the SpeedWorkers interface look like?

Because SpeedWorkers is a "set it and forget it" solution, the interface is focused on providing you with reporting tools to understand how well the solution is working, rather than giving you data about things you need to act on.

The overview report

The SpeedWorkers overview screen allows you to see two of the most important metrics you'll want to monitor to understand the impact dynamic rendering is having on your site:

1. How much faster your content is being delivered to bots
2. How long it takes before content is updated (re-cached)

Quality checks and controls

We also have a quality checks and controls dashboard. These reports monitor for two types of issues:

1. Issues that prevent pages from being delivered altogether (i.e. pass/fail)
2. Issues with individual scripts

Most pre-rendering solutions don't give users much, if any, insight into how well the dynamic rendering is working, but that's not something you have to worry about with SpeedWorkers. You'll get both high-level and granular insight into how everything is being rendered on a daily basis.

Additionally, these quality controls are completely customizable based on your needs. For example, if something isn't present and isn't working properly, we could fall back to doing a live rendering of the page instead.

Site cache monitoring

If everything looks good on your overview page, you can be reassured that SpeedWorkers is doing your job and you don't need to do any further digging, but if you did want to dig further, there are some additional reports like this Site Cache report, which gives you a breakdown of fails versus warnings, as well as the status codes of all cached content.

Delivery metrics

The delivery page provides stats and metrics about how often your dynamic rendered content is being requested by search engines.

Again, this is just another example of how much information you'll be able to get about how well your dynamic rendering is working. Our solution is robust, and we're confident that you'll get consistently great performance out of it, but we don't just want you to take our word for it. That's why we've provided such robust reporting around SpeedWorkers' performance and impact.

A full list of cached and requested pages

And finally, the SpeedWorkers interface gives you a complete list of every page that's been cached and requested.

You have the ability to sort these pages by any metric, such as delivery time or HTTP code. You can even see what each cached page looks like by clicking the camera icon next to the URL.

What results does SpeedWorkers produce for customers?

It's still early, but our customers are already seeing great results from SpeedWorkers. Just how great? Well, it's not uncommon for our customers to see 10-30x improvements in speed!

We've seen even higher though -- for example, pages that were taking 10+ seconds to fully render, after SpeedWorkers, served in 300-400ms.

Overall, our customers can expect SpeedWorkers to:

• Significantly improve their speed-related crawl budget issues
• Solve their "partial indexing" issues by ensuring no JavaScript-loaded content is missed by search engines
• Solve the bot problem with no development work required, keeping your engineering team focused on their user/customer improvements
• Save money by only paying to render content for bots, rather than bots and visitors
• Provide quick wins for increasing revenue from the organic search channel

If you have questions about SpeedWorkers or are interested in learning more about how it could benefit your performance, reach out and let us know! We'd love to hear from you.