Reddit can be a valuable source for information from real people, which is why Google is spending millions on a deal with the platform. But, now, Reddit has started to block many of its results from showing in other search engines properly.
In February of this year Google announced a new deal with Reddit which would see Reddit data used for training Google’s AI models as well as more prominently showing Reddit results within Google Search. Bloomberg reported that the deal was for around $60 million. In the time since, Reddit has also been showing in Google Search drastically more, often outranking websites that Reddit posts are linking to.
Now, Reddit results in other search engines are effectively being blocked.
This behavior was first reported by 404 Media, which notes that Reddit has updated its robots.txt file to block all bots from scraping any part of the site. In the file, Reddit says:
Reddit believes in an open internet, but not the misuse of public content.
In June, just prior to when the file was first updated, Reddit explained that the change was being made following “an uptick in obviously commercial entities who scrape Reddit” and “use Reddit content for any use case they want.” While it’s not said explicitly, it’s also pretty clear that AI training is a clear focus for this change in policy.
But, as a result, search engines other than Google are now not able to show proper results from Reddit as they previously did.
Update 7/25: Speaking to 9to5Google on background, a Reddit representative explains that the problems in other search engines is “not at all” related to the Google partnership, and is a result of the changes to its robots.txt file which are aimed at “all crawlers” that are not willing to commit to not using Reddit data for AI training. Reddit says that the Internet Archie and reddit4research are two examples of crawlers that continue to work.
Reddit is “open” to working with others around data crawling and is in discussions with “multiple” search engines, but has not reached deals with all of them due to promises around how Reddit content would be used, including in the training of AI.
Our original coverage follows (and our headline has been updated alongside this additional context from Reddit):
404 notes that Bing, DuckDuckGo, Mojeek, and Qwant are all affected, with results either not showing anything recent, or not showing the full site result. Kagi, a paid search engine, is apparently still showing data, but only because it buys some of its search index from Google, which continues to have access to Reddit data through the aforementioned deal.
FTC: We use income earning auto affiliate links. More.
Comments