AI Maggots: Are These Creatures Killing the Web?
September 18, 2025
The short answer is, “Yep.”
The early days of the free, open Web held such promise. Alas, AI is changing the Internet and there is, apparently, nothing we can do about it. The Register laments, “AI Web Crawlers Are Destroying Websites in their Never-Ending Hunger for Any and All Content: But the Cure May Ruin The Web.…” Writer Steven J. Vaughan-Nichols tells us a whopping 30% of traffic is now bots, according to Cloudflare. And 80% of that, reports Fastly, comes from AI-data fetcher bots. Web crawlers have been around since 1993, of course, but this volume is something new. And destructive. Vaughan-Nichols writes:
“Fastly warns that [today’s AI crawlers are] causing ‘performance degradation, service disruption, and increased operational costs.’ Why? Because they’re hammering websites with traffic spikes that can reach up to ten or even twenty times normal levels within minutes. Moreover, AI crawlers are much more aggressive than standard crawlers. As the InMotionhosting web hosting company notes, they also tend to disregard crawl delays or bandwidth-saving guidelines and extract full page text, and sometimes attempt to follow dynamic links or scripts. he result? If you’re using a shared server for your website, as many small businesses do, even if your site isn’t being shaken down for content, other sites on the same hardware with the same Internet pipe may be getting hit. This means your site’s performance drops through the floor even if an AI crawler isn’t raiding your website. Smaller sites, like my own Practical Tech, get slammed to the point where they’re simply knocked out of service. Thanks to Cloudflare Distributed Denial of Service (DDoS) protection, my microsite can shrug off DDoS attacks. AI bot attacks – and let’s face it, they are attacks – not so much.”
Even big websites are shelling out for more processor, memory, and network resources to counter the slowdown. And no wonder: According to Web hosting firms, most visitors abandon a site that takes more than three seconds to load. Site owners have some tools to try mounting a defense, like paywalls, logins, and annoying CAPTCHA games. Unfortunately, AI is good at getting around all of those. As for the tried and true, honor-system based robots.txt files, most AI crawlers breeze right on by. Hey, love maggots.
Cynthia Murrell, September 18, 2025
Comments
Got something to say?
-
- Subscribe to Beyond Search
Feature archive
News archive
-
