Following up on my last post regarding AI’s impact on blogs, I came across an interesting new option that Cloudflare is rolling out. Instead of facing the dilemma of leaving content open to AI or fully blocking its access, Cloudflare is offering a “pay-per-crawl” option.
We believe your choice need not be binary — there should be a third, more nuanced option: You can charge for access. Instead of a blanket block or uncompensated open access, we want to empower content owners to monetize their content at Internet scale.
For this third option, Cloudflare does the following:
Pay per crawl grants domain owners full control over their monetization strategy. They can define a flat, per-request price across their entire site. Publishers will then have three distinct options for a crawler:
Block: Deny access entirely, with no option to pay.
Allow: Grant the crawler free access to content.
Charge: Require payment at the configured, domain-wide price.
Check out the article for more technical details. It has a bunch of information like examples, how it functions today, and how it could function in the future. It’s still in private beta, but I’m interested to see how well it will catch on.
I hope everyone in America had a safe 4th of July!
Thanks for reading!
Chad Callihan 