Indexing the Internet is resource-intensive work. With more than half a million new websites being created globally every day, search engines are in constant pursuit of more efficient crawling and indexing methods. (That’s how the sitemap was born almost twenty years ago.) With so much content to crawl, it can take days or even weeks for search engines to pick up on new, updated, or deleted pages, costing businesses untold sums in lost revenue.
Thankfully, these kinds of delays may soon be behind us. In October 2021, Microsoft Bing and Yandex announced IndexNow, an open protocol that allows websites to actively notify participating search engines of changes to their content. Instead of crawling entire websites to test for changes, search engines will know exactly which pages to prioritize in crawl.
Traditional crawling no longer makes sense for the modern web.
Stan Chauvin, Botify Co-founder and Co-Chief Innovation Officer
Microsoft calls IndexNow “an initiative for a more efficient Internet.” Not only does it promise a major reduction in energy costs, but it also lightens the workload for site owners and ensures the timely delivery of their content. As part of its ongoing commitment to this efficiency initiative, Botify’s enterprise organic search platform will support IndexNow integration by the end of Q2 this year.
Botify’s partnership with Bing dates back to 2019, when Bing was piloting its content submission API. Unlike IndexNow, the API takes Bingbot out of the indexing equation entirely. Instead of notifying search engines of pages to prioritize in crawl, it pushes those changes directly to the index.
Site owners were eager to take advantage of this new tech, but managing the API calls proved complex. That’s why Botify streamlined the process for SEOs with FastIndex, a component of Botify Activation. Using crawl data from SpeedWorkers, FastIndex pushes site changes directly to Bing’s index while also keeping customer sitemaps meticulously updated for easy indexing by other search engines.
But Microsoft is changing the game yet again by opening up the IndexNow protocol to all search engines. Google recently announced that it would begin testing the protocol. Should it choose to adopt IndexNow, it could have profound implications for SEO globally.
IndexNow is ultimately about conserving resources as the Internet grows ever more massive and complex. Search engines already struggle with resource scarcity, which leaves them no choice but to impose limitations like crawl budget on site owners.
Botify Co-Chief Innovation Officer and Co-founder Stan Chauvin sees innovation toward search efficiency as inevitable.
“Traditional crawling no longer makes sense for the modern web,” he says. “It’s way too expensive. That’s why all our products function as interfaces between websites and search bots: to streamline the transfer of information.”
When FastIndex launches its IndexNow integration in Q2, Botify will automatically notify all participating search engines of updates, deletions, and additions to customer sites. Users will also still have access to existing FastIndex features, including the Bing content submission API and automated sitemap management.
Several Botify partners – including Cloudflare, Akamai, Yext, and WordPress, among others – have already joined the IndexNow initiative, accelerating the transition toward a more efficient Internet.
To learn more about Botify, get in touch, or subscribe to our blog to receive monthly updates in your inbox! We solemnly swear we are up to no spam.
