Google Reveals How Websites 'Disappeared' From Its Search Engine

In today’s time, the worst nightmare of a website is not showing up in Google Search results. Google is mostly the internet for many people, in fact, the whole SEO industry we have right now is built around Google’s search engine.

The sad dream became true for some websites earlier this year when a nasty bug deindexed web pages from the Google Search database.

In a new blog post, Google now explains how it “temporarily lost part of the Search index” which caused websites to disappear from its Search engine.

Given the trillions of pages Google has to index, it keeps the data across different data centers. So, it becomes a tedious task to update the database on each of them when Google needs to the show latest version of the index to the end-users at the same time.

It is a gradual process which could take several days in case some sensitive pieces of infrastructure need to be updated. During such an update, Google’s deployment system broke while trying to add changes to the Search index on April 5.

“As we were updating the index over some of our data centers, a small number of documents ended up being dropped from the index accidentally.”

It is estimated that around 4% of Google’s index was affected by the bug. However, the bug didn’t affect every site in the database. The number of pages deindexed could be different for different websites.

Attempts to fix the Search index bug gave rise to another problem. It messed up the data and reports that show in the Google Search Console — a platform used by webmasters to analyze their website traffic. Eventually, the inconsistencies in the Search Console database were fixed and on April 28 Google announced that it was able to roll back all of its data centers in just a few hours. But it wasn’t until April 11 that the company officially confirmed that all of its data centers were fully restored.

Google says it has made changes to how it reports bugs and informs concerned webmasters as it tries to issue fixes. The company will now be quicker to share required information through various means, such as via social media channels and by making it easily visible in the Search Console itself.

The search giant got to test its new communication strategy when another bug popped-up on May 22. After a planned upgrade, Google’s duplicate management system ran out memory and was unable to process all incoming URLs.