Reddit was revealed to have blocked multiple search engines and AI crawlers, and officials said it had nothing to do with Google's cooperation

News from this site on July 25. In February this year, Google announced a new agreement with Reddit. Reddit data will be used to train Google’s AI model and highlight Reddit results in Google search results. Bloomberg reports the deal is worth about $60 million. Since then, Reddit has also seen a significant increase in impressions on Google searches. The latest news is that Reddit has banned other search engines from retrieving its website content. Foreign media 404 Media first reported this behavior and pointed out that Reddit has updated its robots.txt file (this site’s note: the agreement file between the website and the crawler) to prevent all robots from crawling any content on the site. This site inquired about the robots.txt file of the Reddit website and learned that the platform stated that "Reddit believes in an open Internet, but does not believe in the abuse of public content." Its "Disallow" option is set to "/", which means that retrieval of content in the root directory of the website is prohibited. All files.

The document was first updated in June of this year, with Reddit explaining that the change was due to "a significant increase in commercial entities scraping Reddit content and misuse of the content."

Foreign media 9To5Google said that search engines such as Bing, DuckDuckGo, Mojeek and Qwant have all been affected. When searching for content using "site:reddit.com", either the latest content or complete website results are not displayed. .

However, in the latest test of this site, Bing and DuckDuckGo have re-displayed the search content of "site:reddit.com".

Reddit spokesperson Tim Rathschmidt said in a statement to The Verge:

“This is completely unrelated to our recent partnership with Google .

We have been in discussions with multiple search engines, but have been unable to Come to terms with everyone because some people are unable or unwilling to make enforceable commitments regarding their use of Reddit content, including their use of AI training.”

The above is the detailed content of Reddit was revealed to have blocked multiple search engines and AI crawlers, and officials said it had nothing to do with Google's cooperation. For more information, please follow other related articles on the PHP Chinese website!