Cloudflare announced new tools Monday that it claims will help end the era of endless AI scraping by giving all sites on its network the power to block bots in one click.
That will help stop the firehose of unrestricted AI scraping, but, perhaps even more intriguing to content creators everywhere, Cloudflare says it will also make it easier to identify which content that bots scan most, so that sites can eventually wall off access and charge bots to scrape their most valuable content. To pave the way for that future, Cloudflare is also creating a marketplace for all sites to negotiate content deals based on more granular AI audits of their sites.
These tools, Cloudflare's blog said, give content creators "for the first time" ways "to quickly and easily understand how AI model providers are using their content, and then take control of whether and how the models are able to access it."
That's necessary for content creators because the rise of generative AI has made it harder to value their content, Cloudflare suggested in a longer blog explaining the tools.
Previously, sites could distinguish between approving access to helpful bots that drive traffic, like search engine crawlers, and denying access to bad bots that try to take down sites or scrape sensitive or competitive data.
But now, "Large Language Models (LLMs) and other generative tools created a murkier third category" of bots, Cloudflare said, that don't perfectly fit in either category. They don't "necessarily drive traffic" like a good bot, but they also don't try to steal sensitive data like a bad bot, so many site operators don't have a clear way to think about the "value exchange" of allowing AI scraping, Cloudflare said.
That's a problem because enabling all scraping could hurt content creators in the long run, Cloudflare predicted.