Posted on by 
The Botify Team
JavaScript is everywhere on the modern web, but it wasn’t always this way.
We recently spoke with Martin Splitt, who explained that the web was originally a document platform composed entirely of HTML.
Over time though, websites evolved from pure information to application through the use of JavaScript. However, JavaScript doesn’t always just add interactivity to pages. It can also contain information.
Because JavaScript can contain information, Google and other search engines render it in order to fully understand our pages. However, that doesn’t always happen successfully or immediately (see: render budget), and it can also add seconds of load time per page, introducing speed-related crawl budget issues for large sites.
Some organizations have attempted to solve for this issue by implementing server-side or pre-rendering solutions, but those can be incredibly resource-intensive and not always reliable.
That’s why we built SpeedWorkers.
Rather watch than read? No problem! Request your copy of our SpeedWorkers video explainer here.
SpeedWorkers is a dynamic rendering solution that sends fully-rendered versions of your pages to search engine bots.
💡 Dynamic Rendering sends fully-rendered content to search engines while serving human visitors with normal, client-side rendered content. It’s pre-rendering, but just for search engine bots. Learn more about dynamic rendering.
This provides customers with two main benefits:
💡 Crawl budget is the number of pages on your site that Googlebot can and is willing to crawl in a given session. Every site’s crawl budget is different, and is determined by factors such as how fast your pages load, how many errors your pages serve, the popularity of your pages, and more. Learn more about crawl budget.
Dynamic rendering is a legitimate, Google-approved solution to these problems that does not constitute cloaking, as Martin reassured us during our dynamic rendering webinar.
Our promise with SpeedWorkers is to:
💡 Need some more background on JavaScript before diving in further? Learn more about how JavaScript works, the difference between client-side and server-side rendered content, and how it all affects SEO in JavaScript 101 for SEOs.
SpeedWorkers operates at the CDN (content delivery network) level.
💡 A content delivery network (CDN) is a network of distributed servers that can help minimize loading delays by reducing the physical distance between the user requesting the page and the server sending the page’s contents.
This allows us to read, render, and cache your CDN content on Botify’s servers continually, and at a rate you can customize.
This equips Botify with an up-to-date collection of all your site’s content, fully-rendered, without you or your CDN needing to do anything on a regular basis.
Being at the CDN level allows SpeedWorkers to intercept bot requests and deliver high-speed, bot-optimized content from Botify’s servers in response.
We went this route with SpeedWorkers for two main reasons. For one, CDNs are extremely close to search engine requests, so we can react very quickly. Second, CDNs are very used to supporting third party applications.
The first iteration of SpeedWorkers is compatible with two of the most popular CDNs: Cloudflare and Akamai. If you use another CDN but are still interested in SpeedWorkers, just let us know! We’re already in the process of adding support for additional CDNs and would love to know what you use so we can prioritize that.
This solution means human users are not affected at all. They will continue to receive the normal, client-side rendered content. Their requests for your content never even go to Botify. They’re processed normally by your CDN.
Since SpeedWorkers, like other dynamic rendering solutions, only pre-renders content for bots, you may be wondering why you wouldn’t also take the burden of rendering JavaScript off your human visitors as well.
There are a few reasons for this:
We want to make our website experiences enjoyable for people, but at the same time, people aren’t the ones that crawl and index the web. Search engines do. And that’s why it’s important to optimize for bots.
💡 Because SpeedWorkers focuses on providing fully-rendered content to search engines, it would not improve your Core Web Vitals metrics, which measure real user interactions with your web pages. Learn more about Core Web Vitals.
The Botify platform comprises three suites:
SpeedWorkers falls under Botify Activation.
But even though SpeedWorkers sits within the Botify Activation suite, it integrates and interacts with Botify Analytics and Botify Intelligence as well.
For example:
Because SpeedWorkers is a “set it and forget it” solution, the interface is focused on providing you with reporting tools to understand how well the solution is working, rather than giving you data about things you need to act on.
The SpeedWorkers overview screen allows you to see two of the most important metrics you’ll want to monitor to understand the impact dynamic rendering is having on your site:
We also have a quality checks and controls dashboard. These reports monitor for two types of issues:
Most pre-rendering solutions don’t give users much, if any, insight into how well the dynamic rendering is working, but that’s not something you have to worry about with SpeedWorkers. You’ll get both high-level and granular insight into how everything is being rendered on a daily basis.
Additionally, these quality controls are completely customizable based on your needs. For example, if something isn’t present and isn’t working properly, we could fall back to doing a live rendering of the page instead.
If everything looks good on your overview page, you can be reassured that SpeedWorkers is doing your job and you don’t need to do any further digging, but if you did want to dig further, there are some additional reports like this Site Cache report, which gives you a breakdown of fails versus warnings, as well as the status codes of all cached content.
The delivery page provides stats and metrics about how often your dynamic rendered content is being requested by search engines.
Again, this is just another example of how much information you’ll be able to get about how well your dynamic rendering is working. Our solution is robust, and we’re confident that you’ll get consistently great performance out of it, but we don’t just want you to take our word for it. That’s why we’ve provided such robust reporting around SpeedWorkers’ performance and impact.
And finally, the SpeedWorkers interface gives you a complete list of every page that’s been cached and requested.
You have the ability to sort these pages by any metric, such as delivery time or HTTP code. You can even see what each cached page looks like by clicking the camera icon next to the URL.
It’s still early, but our customers are already seeing great results from SpeedWorkers. Just how great? Well, it’s not uncommon for our customers to see 10-30x improvements in speed!
We’ve seen even higher though — for example, pages that were taking 10+ seconds to fully render, after SpeedWorkers, served in 300-400ms.
Overall, our customers can expect SpeedWorkers to:
If you have questions about SpeedWorkers or are interested in learning more about how it could benefit your performance, reach out and let us know! We’d love to hear from you.
