Google. It’s the most widely used search engine, and for many, it’s the first thing people type into their web browsers every day. However, it’s not the only search engine people use. According to Alexa, Baidu is the fourth most popular website (with Google ranking #1). In fact, China has the third largest internet user population (731 million back in 2016), so it’s no surprise that Baidu ranks so high.
It’s not uncommon for websites (especially large, enterprise websites) to have an international audience base, so it’s critical for anyone thinking about SEO to be aware of all the major search engines.
With varying features and capabilities, each search engine crawler is configured to work at an entirely different pace and aptitude, oftentimes with a completely different end target (mobile-friendliness, website authority, and so on).
So, when you’re optimizing for an audience in Russia, for example, you’ve got to think like the Russian search engines do (in this case, Yandex!).
A big-picture look at search engines across the globe
Google: the search engine with majority market share
The graph below shows the URLs crawled by Google on a particular website. To illustrate the differences between each search engine and its capabilities, all of the graphs represent the same website over the same period of time, with a day over day view.
In the example, Google crawls close to 1M or 2M per day, 100K-500K of which are new URLs. Its crawl volume exceeds the number of crawled URLs, meaning that your pages can be crawled multiple times in a day (Woohoo!). You can check how often Google is crawling your pages in Google Search Console, or directly in the Botify platform.
How do you ensure Google crawls and indexes your new pages?
There are a few ways that you can go about increasing your crawl rate to ensure Google crawls and indexes your new pages. For example, you can improve your site’s load time (and, as a result, enhance your UX — a win, win!) and submit an updated sitemap making sure to list your most important, new URLs first.
After you’ve taken steps to direct Google to crawl your new URLs, you can use a tool like the Botify Log Analyzer to see if that’s actually happening (e.g. Is Google really crawling my URLs? How often?).
These aspects and more will help you identify if and when your new pages will be crawled and help steer bots in the right direction.
Bing: Microsoft’s search engine
Microsoft launched Bing in 2009, and as of 2017, claimed it enjoyed about 9% global market share and 33% in the U.S.
While similar to Google, Bing has key differences in the way it crawls, indexes, and ranks web pages.
Key differences between Google and Bing
- Ranking: Bing’s ranking algorithms use some of the same factors as Google, like backlinks and mobile-friendliness, but weights them differently than Google — backlinks appear to be much more important for ranking in Bing than Google, while mobile-friendliness is more important in Google than Bing.
- Crawl Prioritization: Bingbot prioritizes URLs by looking at internal linking and sitemaps, while Googlebot uses a multitude of other factors. Additionally, Bingbot sometimes falls into “crawl traps” like faceted navigation and internal search URLs.
- Crawl Volume: As depicted below, Bing’s crawl volume nearly matches the amount of crawled URLs (meaning that, instead of crawling URLs multiple times, it crawls them just once, or much less frequently). Meanwhile, the amount of new URLs crawled remains very low, making it difficult to get new, important pages indexed fast.
But we’ve got big news!
At our recent Crawl2Convert conference in New York, we were thrilled to have Fabrice Canel, Bing’s Principal Program Manager, present the new Bing API that will help you get your newest pages indexed instantaneously. Even better, the API will be directly integrated into the Botify platform! This integration will launch in correspondence with FastIndex, a component of Botify Activation that will allow Botify customers to create fully optimized sitemaps. You’ll be able to increase your daily submission limits and get your pages indexed fast – through FastIndex and the Bing API.
Baidu: China’s main search engine
Baidu is the dominant search engine used in China. It is similar in size and scale to Google in the U.S., and also resembles Google in its investments — for example, just like Google, Baidu has invested in machine learning to improve its ranking algorithms.
In contrast though, Baidu has a huge capacity for crawling. Rather than relying on signals like canonicals to prioritize crawling and indexing, Baidu typically can crawl everything and decide on its own what should be indexed.
Additionally, China heavily prioritizes mobile, so Baidu mobile search results can vary drastically from Baidu desktop search results. Mobile is such a priority in China that Baidu even creates mobile versions of desktop-only websites on the fly.
Baidu also has an initiative similar to Google’s AMP. Baidu’s AMP-like initiative began simply as MIP (Mobile Instant Pages), followed by the introduction of a new, more utilitarian platform called Bear Paw (named in honor of the Baidu logo). In 2018, Baidu announced it would send 50% of its mobile traffic to BP pages.
P.S. Although Google China does exist, it was blocked from mainland China in 2010 and is only available in Hong Kong. So, if your website has a large visitor base from China, they’re likely searching and finding you via Baidu!
Yandex: Russia’s main search engine
Like Baidu, Yandex generally does not crawl URLs more than once in a given timeframe. However, it crawls a fair volume of new URLs (at the rate of nearly 50K per day!), which is optimistic for websites with new web pages that require indexation.
Naver: South Korea’s main search engine
Naver is the most popular search engine in South Korea. This search engine behaves more like an early 2000s Yahoo! – acting as a “front page of the internet.” While they do index content from websites, a lot of their content and pages are curated search pages rather than standard search results. Naver connects users based on shared interests and uses this as a means of funnelling relevant information to them. Naver also launched Knowledge-IN as a public Q&A portal for users to get their questions answered and for Naver to collect data in order to prioritize topics according to popularity.
In some ways, Naver is more publisher than search engine. Ranking depends heavily on their determination of how authoritative a website is, using both human feedback and a ranking algorithm. Social media presence also plays a role in the algorithm.
We sure can!
So, while you might use a single search engine (likely Google), remember that there are millions of others around the world who use different engines to search the web. Different search engines mean different experiences and different things to keep in mind for SEO. For example, Naver search results are fueled by communities and interests, Baidu prioritizes optimized mobile experiences.
Knowing what search engines your audience uses is also important for staying up-to-date on the latest news from that search engine. When the next BERT comes along, for example, you’ll want to know about it so you can keep pace and avoid falling behind.
While SEO may not be the same across the globe, it’s equally important everywhere. So, put your website to the test and make sure you do all that you can to reach the people that are looking for you, no matter where they are!