[Update: Google statement] Study finds that Google varies search results even when users are logged out or browsing incognito

Ben Schoon | Dec 4 2018 - 7:11 am PT

It’s no secret that Google holds a lot of data about its users, and that includes personalized search results based on a variety of factors. Thanks to a new study, we’re getting some further insight into how search results are personalized, as well as how being logged out affects matters.

Update: Google statement added.

One of Google Search’s larger competitors, DuckDuckGo, conducted a study recently that looked into the “filter bubble” which Google applies to search. This “bubble” affects Google search results with personalized content which is based on things such as your account activity, search history, and much more. A common way users think they can avoid this is by logging out of their account or simply browsing incognito, but apparently, that doesn’t do the trick.

This study, conducted in June of 2018, reveals that despite efforts being taken to avoid the “filter bubble,” users still see personalized search results on Google. Obviously, DuckDuckGo is a biased source for this study seeing as its product is a privacy-focused search engine. However, that doesn’t take away from many of the findings here.

One of the more interesting findings in this study was how search results vary even when users are logged out of a Google account and browsing in a private browsing mode (incognito). The study took three keywords, “gun control,” “immigration,” and “vaccinations” to pull these results with 76 participants. Users tested this in “normal” mode which was logged into a Google account, as well as logged out and in private browsing mode.

To summarize the results, every single test delivered mainly unique results with each keyword regardless of where it was searched. “Gun control” saw 52/76 with unique results when in private mode and 45/76 when logged in. Immigration showed 43/76 as unique when logged out and 48/76 unique when logged in. “Vaccinations” actually had the exact same totals with 70/76 participants seeing unique results in either scenario.

Clearly, Google still has personalized search results even when users are seemingly outside of the “filter bubble.” Of course, search results can still vary based on time, location, and a few other factors, but DuckDuckGo actually eliminated the time variable by searching at the same time. Further, local results were at a minimum during these searches.

More tests in this study also found that Google simply didn’t include some results in searches for the same keyword. Most participants saw some variation of a similar list of 10 domains, but others saw more or less, as well as seeing totally different orders to these results as illustrated below.

It’s also interesting, and further drives home the first point from the study, that Google barely makes changes to a user’s search results when switching to a private browsing mode. The study explains:

We saw that when randomly comparing people’s private modes to each other, there was more than double the variation than when comparing someone’s private mode to their normal mode. We often hear of confusion that private browsing mode enables anonymity on the web, but this finding demonstrates that Google tailors search results regardless of browsing mode.

Google has said in a comment to The Verge that “search results can change by the minute and sometimes even by the second” and that “personalization is done a small fraction of the total number of queries entered into search.” However, the company didn’t comment on if (or why) Google search results are still personalized when logged out/in private browsing. It did mention, though, that it “does not personalize results for incognito searches using signed-in search history.”

Update: Google further provided a statement to 9to5Google. In short, Google sees the study as flawed citing time and location as factors that could have further affected the results.

This study’s methodology and conclusions are flawed since they are based on the assumption that any difference in search results are based on personalization. That is simply not true. In fact, there are a number of factors that can lead to slight differences, including time and location, which this study doesn’t appear to have controlled for effectively.

9to5Google’s Take

You can read DuckDuckGo’s full study for more on the results, but the question really comes down to if this all really matters. That’s a very personal question. This study shows that, when using Google Search, there’s really no way to avoid results being skewed in some way. For some, that’s an unsettling thing to know, especially because it’s seemingly impossible to get a completely “neutral” result.