Billions at Stake: The AI Bot Wars Begin

August 7, 2025

Dino 5 18 25No AI. Just a dinobaby being a dinobaby.

I noticed that the puffs of smoke were actually canon fire in the AI bot wars. The most recent battle pits Cloudflare (a self-declared policeman of the Internet) against Perplexity, one of the big buck AI outfits. What is the fight? Cloudflare believes there is a good way to crawl and obtain publicly accessible content. Perplexity is just doing what those Silicon Valley folks have done for decades: Do stuff and apologize (or not) later.

WinBuzzer’s “Cloudflare Accuses Perplexity of Using ‘Stealth Crawlers’ to Evade Web Standards” said on August 4, 2025, at a time that has not yet appeared on my atomic clock:

Web security giant Cloudflare has accused AI search firm Perplexity of using deceptive “stealth crawlers” to bypass website rules and scrape content. In a report Cloudflare states Perplexity masks its bots with generic browser identities to ignore publisher blocks. Citing a breach of internet trust, Cloudflare has removed Perplexity from its verified bot program and is now actively blocking the behavior. This move marks a major escalation in the fight between AI companies and content creators, placing Perplexity’s aggressive growth strategy under intense scrutiny.

I like the characterization of Cloudflare as a Web security giant. Colorful.

What is the estimable smart software company doing? Work arounds. Using assorted tricks, Perplexity is engaging in what WinBuzzer calls “stealth activity.” The method is a time honored one among some bad actors. The idea is to make it difficult for routine filtering to stop the Perplexity bot from sucking down data.

If you want the details of the spoofs that Perplexity’s wizards have been using, navigate to this Ars Technica post. There is a diagram that makes absolutely crystal clear to everyone in my old age home exactly what Perplexity is doing. (The diagram captures a flow I have seen some nation state actors employ to good effect.)

The part of the WinBuzzer story I liked addressed the issue of “explosive growth and ethical scrutiny.” The idea of “growth” is interesting. From my point of view, the growth is in the amount of cash that Perplexity and other AI outfits are burning. The idea is, “By golly, we can make this AI stuff generate oodles of cash.” The ethical part is a puzzler. Suddenly Silicon Valley-type AI companies are into ethics. Remarkable.

I wish to share several thoughts:

  1. I love the gatekeeper role of the “Web security giant.” Aren’t commercial gatekeepers the obvious way to regulate smart software? I am not sharing my viewpoint. I suggest you formulate your own opinion and do with it what you will.
  2. The behavior of Perplexity, if the allegations are accurate, is not particularly surprising. In fact, in my opinion it is SOP or standard operating procedure for many companies. It is easier to apologize than ask for permission. Does that sound familiar? It should. From Google to the most recent start up, that’s how many of the tech savvy operate. Is change afoot? Yeah, sure. Right away, chief.
  3. The motivation for the behavior is pragmatic. Outfits like Perplexity have to pull a rabbit out of the hat to make a profit from the computational runaway fusion reactor that is the cost of AI. The fix is to get more content and burn more resources. Very sharp thinking, eh?

Net net: I predict more intense AI fighting. Who will win? The outfits with the most money. Isn’t that the one true way of the corporate world in the US in 2025?

Stephen E Arnold, August 7, 2025

Comments

Got something to say?





  • Archives

  • Recent Posts

  • Meta