Get To The Bottom Of Crawl Budget Issues With SpeedWorkers

Search engines like Google don't have unlimited resources to crawl and render every page on every website. Instead, they compromise and allot a specific amount of time indexing your website, known as crawl budget.

JavaScript makes this even more challenging for search engines. Love it or hate it, JavaScript is on nearly every website on the modern web. It allows you to create unique and interactive experiences, contains important content for users, and can even run directly within your browser to reduce the demand on your servers. But this comes at a cost.

While these websites look great for humans, there is a lot more going on behind the scenes, making it difficult for non-human users like Google to crawl and render pages.

Before search engines can index your pages, bots first need to render all of the JavaScript on the page. This process takes time. In some cases, it can be too difficult for search engines to understand if your page is essential, especially for large websites. This becomes a systemic issue, and search engines will actively de-prioritize and ignore pages with lots of JavaScript. Simply put, pages with JavaScript need to be optimized to maximize your crawl budget. So how do we do that?

SpeedWorkers, the best-in-class, enterprise-grade dynamic rendering solution that makes it faster for bots to crawl and render your pages, can help. SpeedWorkers sends fully-rendered versions of your pages to search engines so that more of your website gets crawled by bots, indexed by search engines, and discovered by your users.

What's New in SpeedWorkers?

To make enterprise dynamic rendering plausible, the devils are in the details. In our newest release, we've added a new URL inventory selector, cache behaviors dashboard, and quality control rule builder to make one of the most nuanced aspects of Botify Activation more transparent and easier to manage. Enterprise websites have millions of pages and need granular control over what, when, and how pages are dynamically rendered for bots, as well as additional quality control tools to test what's happening behind the scenes at scale. If this is your first time learning about SpeedWorkers, great! We highly recommend you check out our introductory blog to SpeedWorkers first. If not, let's jump in.

Choose What Pages Are Cached

When SpeedWorkers first launched, it required coordination work to determine which pages it should run on. Today, you can do this directly within the platform. SpeedWorkers now lets you choose which pages you want to be cached on your website. Simply select the list of URLs, or choose to automatically add URLs each time a bot requests a page from your website to ensure you never miss a page. You can also define which type of bot requests are important to react to or ignore such as Google, Baidu, or all bots. We're also hard at work adding other sources like the Shopify and Google Search Console API to ensure all new pages are cached instantly.

SpeedWorkers also allows you to automatically define and exclude marketing or tracking parameters so that only canonicals are being cached and delivered to search engines so you're not wasting resources caching content that bots don't need. Simply create rules to block unimportant directories or extensions such as subdomains like mobile.botify.com or URLs that end with .txt

Define Page Cache Behavior

All pages on your website have different levels of importance regarding how frequently they need to be cached. Your About page probably isn't changing too often and won't need to be updated, while strategic pages like product pages or news articles may be updated frequently and will need to be cached multiple times a day. Cache behaviors give you the control and confidence to render pages efficiently with different caching rates and rules for different types of content so that you're never wasting resources on pages that shouldn't be re-rendered. Simply create default cache rules for your website, and add additional cache behavior rules for groups of pages that need to be treated differently than the rest of your website.

When creating custom cache rules, you can create different behaviors for groups of pages to adjust cache lifetime to make sure search engines always receive fresh content. You may also have specific segments of your website where you want that behavior to be a bit different. Having this type of control on the pages you want SpeedWorkers to render is critical for your bottom line. Why waste server resources for pages that shouldn't be rendered?

Set Controls To Ensure Your Strategic Pages Are Properly Rendered

JavaScript is notorious for rendering incorrectly, and for enterprise websites, you need to be able to do this quickly and at scale. SpeedWorkers allows you to build and run custom quality control tests to ensure interactive and third-party content is indexable by search engines with just a few clicks. Create and run quality checks with custom targets, thresholds, and fallbacks for your strategic pages to ensure the JavaScript content on your pages is properly rendered before they are delivered.

What Results Does SpeedWorkers Produce For You?

Since we released SpeedWorkers last year, customers have already seen great results. Just how great? Well, it's not uncommon for our customers to see 10-40x improvements in speed!

We've seen even higher though -- for example, pages that were taking 10+ seconds to render fully, after SpeedWorkers, served in 300-400ms (All of this is happening turn-key).

Overall, our customers can expect SpeedWorkers to:

• Significantly improve their speed-related crawl budget issues
• Solve their "partial indexing" issues by ensuring no JavaScript-loaded content is missed by search engines
• Solve the bot problem with no development work required, keeping your engineering team focused on their user/customer improvements
• Eliminate the impact of bots traffic on server load
• Provide quick wins for increasing revenue from the organic search channel

If you're ready to get serious about dynamic rendering and have questions about SpeedWorkers or are interested in learning more about how your website performance can benefit, let us know! We'd love to hear from you.