Internet security firm Cloudflare announces a novel solution to stifle the activity of artificial intelligence (AI) bots and data scrapers on their websites to 'preserve a safe internet.”

The solution, described as an “easy button” designed to prevent bot access to web content, comes at a time when creators are accusing AI companies

Cloudflare, a company known for providing internet security services, has recently announced a new feature designed to help users easily block AI bots and data scrapers from accessing their websites. This move comes amid ongoing concerns raised by content creators, who have accused AI companies of using their proprietary data to train AI models without seeking express permission.

According to Cloudflare, the new feature will be available to users across all tiers, including those on the free plan. To activate the feature, users can simply navigate to the Security section on their Cloudflare dashboard and toggle between their selected preferences.

“We hear clearly that customers don’t want AI bots visiting their websites, and especially those that do so dishonestly,” the announcement reads. “To help, we’ve added a brand new one-click to block all AI bots.”

In 2023, Cloudflare introduced a feature to enable customers to block AI bots that don’t adhere to the rules, such as seeking consent before using licensed data to train their models. However, Cloudflare noted in its post that despite the clarification, an overwhelming majority of users still opted to block the bots from accessing their websites.

Instead of complying with the rules, Cloudflare’s data reveals a growing number of bots that still attempt to bypass guardrails designed to keep them out. The bots typically crawl websites by leaning on a false user agent to mislead security measures, but Cloudflare says a year’s worth of monitoring has shed significant insights.

“Sadly, we’ve observed bot operators attempt to appear as though they are a real browser by using a spoofed user agent,” the post reads. “We’ve monitored this activity over time, and we’re proud to say that our global machine learning model has always recognized this activity as a bot, even when operators lie about their user agent.”

The firm says it is able to identify AI bots masquerading as real web browsers through several methods, including the use of a bot scoring metric system. Furthermore, Cloudflare says bot attempts to crawl websites at scale leave several glaring fingerprints that are easily identifiable, given the firm’s 57 million requests per second.

Mainstream bots get all the attention

Cloudflare disclosed that users largely contain bot activity from mainstream AI developers, but lesser-known companies are running in the shadows. The analysis indicated that Bytespider, the bot from Bytedance, led other bots in terms of activities by crawling a staggering 40.40% of tracked websites.

Bytespider outperformed OpenAI’s GPTBot and other crawlers by Meta (NASDAQ: META), Anthropic AI and Google (NASDAQ: GOOGL) by a country mile, with the firm pledging to crack down on attempts to bypass guardrails using a combination of emerging technologies.

“We fear that some AI companies intent on circumventing rules to access content will persistently adapt to evade bot detection,” said Cloudflare. “We will continue to keep watch and add more bot blocks to our AI scrapers and crawlers rule and evolve our machine learning models.”

In order for artificial intelligence (AI) to work right within the law and thrive in the face of growing challenges, it needs to integrate an enterprise blockchain system that ensures data input quality and ownership—allowing it to keep data safe while also guaranteeing the immutability of data. Check out CoinGeek’s coverage on this emerging tech to learn more why Enterprise blockchain will be the backbone of AI.

Watch: sCrypt Hackathon students realize there’s more to blockchain

New to blockchain? Check out CoinGeek’s Blockchain for Beginners section, the ultimate resource guide to learn more about blockchain technology.

The above is the detailed content of Internet security firm Cloudflare announces a novel solution to stifle the activity of artificial intelligence (AI) bots and data scrapers on their websites to 'preserve a safe internet.”. For more information, please follow other related articles on the PHP Chinese website!