Search engines like Google don’t have unlimited resources to crawl and render every page on every website. Instead, they compromise and allot a specific amount of time indexing your website, known as crawl budget.
While these websites look great for humans, there is a lot more going on behind the scenes, making it difficult for non-human users like Google to crawl and render pages.
SpeedWorkers, the best-in-class, enterprise-grade dynamic rendering solution that makes it faster for bots to crawl and render your pages, can help. SpeedWorkers sends fully-rendered versions of your pages to search engines so that more of your website gets crawled by bots, indexed by search engines, and discovered by your users.
To make enterprise dynamic rendering plausible, the devils are in the details. In our newest release, we’ve added a new URL inventory selector, cache behaviors dashboard, and quality control rule builder to make one of the most nuanced aspects of Botify Activation more transparent and easier to manage. Enterprise websites have millions of pages and need granular control over what, when, and how pages are dynamically rendered for bots, as well as additional quality control tools to test what’s happening behind the scenes at scale. If this is your first time learning about SpeedWorkers, great! We highly recommend you check out our introductory blog to SpeedWorkers first. If not, let’s jump in.
When SpeedWorkers first launched, it required coordination work to determine which pages it should run on. Today, you can do this directly within the platform. SpeedWorkers now lets you choose which pages you want to be cached on your website. Simply select the list of URLs, or choose to automatically add URLs each time a bot requests a page from your website to ensure you never miss a page. You can also define which type of bot requests are important to react to or ignore such as Google, Baidu, or all bots. We’re also hard at work adding other sources like the Shopify and Google Search Console API to ensure all new pages are cached instantly.
SpeedWorkers also allows you to automatically define and exclude marketing or tracking parameters so that only canonicals are being cached and delivered to search engines so you’re not wasting resources caching content that bots don’t need. Simply create rules to block unimportant directories or extensions such as subdomains like mobile.botify.com or URLs that end with .txt
All pages on your website have different levels of importance regarding how frequently they need to be cached. Your About page probably isn’t changing too often and won’t need to be updated, while strategic pages like product pages or news articles may be updated frequently and will need to be cached multiple times a day. Cache behaviors give you the control and confidence to render pages efficiently with different caching rates and rules for different types of content so that you’re never wasting resources on pages that shouldn’t be re-rendered. Simply create default cache rules for your website, and add additional cache behavior rules for groups of pages that need to be treated differently than the rest of your website.
When creating custom cache rules, you can create different behaviors for groups of pages to adjust cache lifetime to make sure search engines always receive fresh content. You may also have specific segments of your website where you want that behavior to be a bit different. Having this type of control on the pages you want SpeedWorkers to render is critical for your bottom line. Why waste server resources for pages that shouldn’t be rendered?
Since we released SpeedWorkers last year, customers already see great results. Just how great? Well, it’s not uncommon for our customers to see 10-40x improvements in speed!
We’ve seen even higher though — for example, pages that were taking 10+ seconds to render fully, after SpeedWorkers, served in 300-400ms (All of this is happening turn-key).
Overall, our customers can expect SpeedWorkers to:
If you’re ready to get serious about dynamic rendering and have questions about SpeedWorkers or are interested in learning more about how your website performance can benefit, let us know! We’d love to hear from you.