By Stan Chauvin
Bottom line, and much to our chagrin, budgets exist. Most of us just don’t have unlimited resources to spend on whatever we want.
It’s no different for search engines.
Search engines don’t have all the resources and time in the world to crawl everything every day. To solve for this, search engines asked the question, “How can we get the best picture of the web in the cheapest and fastest way possible?”
The answer became crawl budget, which Google’s Gary Illyes defines as “the number of URLs Googlebot can and wants to crawl.”
SEO professionals are pretty familiar with this concept by now. We know that we need to understand what search engines are and aren’t seeing on our websites, and why.
Budgets matter because they affect our bottom line.
If search engines aren’t finding content, the content won’t be indexed, and content that the search engines don’t index can’t rank and earn traffic. When content isn’t visited by searchers, they can’t become customers.
You could also look at this from a search engine’s point of view. If Google and others can’t find and index the valuable content on the web, their search engine will be less relevant, and ultimately less used.
At Botify, we are calling this Render Budget.
This means that some details about your page might be missed. You may have even experienced the negative ramifications of this reality for yourself:
There are three main reasons for this.
This also means that the time it takes for the content to be “ready” is relative. Is some content enough? Is all the content needed? This can differ from page to page, so Googlebot doesn’t always know when it should stop rendering and index the page.
The complexity of this new reality may leave you wanting to run for the hills, but as SEOs, we need to stick with it if we want to stay competitive in the future. After all, for every page of your website that is too hard to render, Google will render three of your competitor’s pages.
But how can we do that?
SEOs learned this well, and mastered the ability to optimize websites for crawling and indexing.
Being effective in this new environment requires a departure from the way SEOs are used to thinking.
For example, we used to measure crawl rate on our websites by analyzing our log files and counting the number of search engines bot hits on our unique URLs.
We used to count the time it took for the server to send the page. This was a totally valid way of thinking when dealing with HTML-only pages that only had two states: loading or loaded.
And last but not least, we used to measure search engines bot activity by adding server-side SEO tracking. This worked fine in HTML-only days when there was only a single request on each page.
If crawl budget is the number of URLs search engines bot can and wants to crawl, render budget is the number of pages search engines bot is able to execute (render). Think about it this way: if crawl budget is search engines opening the envelope, then render budget is search engines reading the letter.
Like crawl budget, render budget acknowledges that search engines simply does not have time to render everything, every day.
As a result, there are two important questions SEOs need to be able to answer:
SEOs should be vigilant to test their URLs to see if search engines have indexed important content. One of the most basic ways to do this is to use search engines’ “site:” advanced search operator.
Just input site:example.com “(snippet of that page’s important content)” into the search bar.
One way to find out how much of your content Google has rendered is to calculate your rendering ratio. This will reveal what percentage of your content Google is truly seeing.
First, find an expression that you know is in the HTML-only version of your pages.
Next, find an expression you know is in the rendered version of your pages. This text should not be present in the initial page response from the server, but rather only present after rendering.
Using the same site: advanced search operator, search for the HTML-only phrase. This will show you how many total pages Google has indexed. Next, search for the HTML-only + rendered phrase. This will show you how many pages Google has rendered and run through its second phase of indexing.
By dividing the number of rendered pages by the number of indexed pages, you’ll arrive at your rendering ratio. In the example below, only about 80% of the pages are being rendered and going through the second phase of indexing.
These complexities certainly make SEO harder, but not impossible.
So how can we do that?
Your development team is your best ally when it comes to grasping how your website behaves, so facilitate an open line of communication with them. Let them know about the SEO impact of their website changes, and be willing to listen to the issues they’re trying to tackle as well.
Many engineers aren’t programmed to think about SEO, so come alongside them and solve these issues together.
It’s important to set appropriate targets in this new landscape. SEOs need to start measuring their success by metrics such as first contentful paint, time to interactive, and the size/number of requests.
SEO has changed, so our measurements for success need to change with it.
When it comes to improving crawl and render budgets on your own websites, we recommend tools such as: