Self-Appointed Gatekeepers and AI Wizards Clash
August 11, 2025
No AI. Just a dinobaby being a dinobaby.
Cloudflare wants to protect those with content. Perplexity wants content. Cloudflare sees an opportunity to put up a Google-type toll booth on the Information Highway. Perplexity sees traffic stops of any type the way a soccer mom perceives an 80 year old driving at the speed limit.
Perplexity has responded to Cloudflare’s words about Perplexity allegedly using techniques to crawl sites which may not want to be indexed.
“Agents or Bots? Making Sense of AI on the Open Web” states:
Cloudflare’s recent blog post managed to get almost everything wrong about how modern AI assistants actually work.
In addition to misunderstanding 20-25M user agent requests are not scrapers, Cloudflare claimed that Perplexity was engaging in “stealth crawling,” using hidden bots and impersonation tactics to bypass website restrictions. But the technical facts tell a different story.
It appears Cloudflare confused Perplexity with 3-6M daily requests of unrelated traffic from BrowserBase, a third-party cloud browser service that Perplexity only occasionally uses for highly specialized tasks (less than 45,000 daily requests).
Because Cloudflare has conveniently obfuscated their methodology and declined to answer questions helping our teams understand, we can only narrow this down to two possible explanations.
- Cloudflare needed a clever publicity moment and we–their own customer–happened to be a useful name to get them one.
- Cloudflare fundamentally misattributed 3-6M daily requests from BrowserBase’s automated browser service to Perplexity, a basic traffic analysis failure that’s particularly embarrassing for a company whose core business is understanding and categorizing web traffic.
The idea is to provide two choices, a technique much-loved by vaudeville comedians on the Paul Whiteman circuit decades ago; for example, Have you stopped stealing office supplies?
I find this situation interesting for several reasons:
- Smart software outfits have been sucking down data
- The legal dust ups, the license fees, even the posture of the US government seems dynamic; that is, uncertain
- Clever people often find themselves tripped by their own clever lines.
My view is that when tech companies squabble, the only winners are the lawyers and the users lose.
Stephen E Arnold, August 11, 2025
Comments
Got something to say?
-
- Subscribe to Beyond Search
Feature archive
News archive
-
