That’s just BattleBots with a different name.
Cloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.
Submitted 4 weeks ago by Tea@programming.dev to technology@lemmy.world
https://blog.cloudflare.com/ai-labyrinth/
Comments
AtomicHotSauce@lemmy.world 4 weeks ago
aviationeast@lemmy.world 4 weeks ago
You’re not wrong.
IrateAnteater@sh.itjust.works 4 weeks ago
Ok, I now need a screensaver that I can tie to a cloudflare instance that visualizes the generated “maze” and a bot’s attempts to get out.
ininewcrow@lemmy.ca 4 weeks ago
They should program the actions and reactions of each system to actual battle bots and then televise the event for our entertainment.
singletona@lemmy.world 4 weeks ago
Then get bored when it devolves into a wedge meta.
supersquirrel@sopuli.xyz 4 weeks ago
No, it is far less environmentally friendly than warrior bots made of metal, plastic, and electronics full of nasty little things like batteries blasting, sawing, burning and smashing one another to pieces.
RelativeArea1@sh.itjust.works 4 weeks ago
this is some fucking stupid situation, we somewhat got a faster internet and these bots messing each other are hugging the bandwidth.
melpomenesclevage@lemmy.dbzer0.com 4 weeks ago
nothing can be improved while capitalism exists; all improvement will be seized and used to oppress.
morrowind@lemmy.ml 4 weeks ago
How can authority not exist? That’s staggeringly broad
IrateAnteater@sh.itjust.works 4 weeks ago
That’s not really relevant here. This is more of a “genie is out of the bottle and now we have to learn how to deal with it situation”. The idea and technology of bots and AI training already exists. There’s no socioeconomic system that is going to magically make that go away.
QuarterSwede@lemmy.world 4 weeks ago
The problem you aren’t recognizing is that, until humans are no longer driven by self preservation, there will always be oppression in any system. They all have and will continue to breakdown. It’s easy to blame capitalism but even socialist systems eventually cave under the weight of greed and power. We are the problem mon frère.
dual_sport_dork@lemmy.world 4 weeks ago
Especially since the solution I cooked up for my site was to identify the incoming requests from these damn bots – which is not difficult, since they ignore all directives and sanity and try to slam your site with like 200+ requests per second, that makes 'em easy to spot – and simply IP ban them.
In fact, anybody who doesn’t exhibit a sane crawl rate gets blocked from my site automatically. For a while, most of them were coming from Russian IP address zones for some reason. These days Amazon is the worst offender, I guess their Rufus AI or whatever the fuck it is tries to pester other retail sites to “learn” about products rather than sticking to its own domain.
Fuck 'em. Route those motherfuckers right to /dev/null.
Buelldozer@lemmy.today 4 weeks ago
and try to slam your site with like 200+ requests per second
Your solution would do nothing to stop the crawlers that are operating 10ish rps. There’s ones out there operating at a mere 2rps but when multiple companies are doing it at the same time 24x7x365 it adds up.
Some incredibly talented people have been battling this since last year and your solution has been tried multiple times. It’s not effective in all instances and can require a LOT of manual intervention and SysAdmin time.
Flagstaff@programming.dev 4 weeks ago
Geez, that’s a lot of requests!
desktop_user@lemmy.blahaj.zone 4 weeks ago
the only problem with that solution being applied to generic websites is schools and institutions can have many legitimate users from one IP address and many sites don’t want a chance to accidentally block one.
morrowind@lemmy.ml 4 weeks ago
Cloudflare offers that too, but you can’t always tell
ininewcrow@lemmy.ca 4 weeks ago
It’s what I’ve been saying about technology for the past decade or two … we’ve hit an upper limit to our technological development … that limit is on individual human greed where small groups of people or massively wealthy people hinder or delay any further development because they’re always trying to find ways to make money off it, prevent others from making money off it, monopolize an area or section of society … capitalism is literally our world’s bottleneck and it’s being choked off by an oddly shaped gold bar at this point.
drmoose@lemmy.world 4 weeks ago
Lol website traffic accounts for like 1% of bandwidth budget. 1 netflix movie is like 20k web pages.
oldfart@lemm.ee 4 weeks ago
So the web is a corporate war zone now and you can choose feudal protection or being attacked from all sides. What a time to be alive.
theparadox@lemmy.world 4 weeks ago
There is also the corpo verified id route. In order to avoid the onslaught of AI bots and all that comes with them you’ll need to sacrifice freedom, anonymity, and privacy like a good little peasant to prove you aren’t a bot… and so will everyone else. You’ll likely be forced to deal with whatever AI bots are forced upon you while within the walls but better an enemy you know I guess?
kandoh@reddthat.com 4 weeks ago
Burning 29 acres of rainforest a day to do nothing
cantstopthesignal@sh.itjust.works 4 weeks ago
Bitcoin?
zovits@lemmy.world 4 weeks ago
It certainly sounds like they generate the fake content once and serve it from cache every time: “Rather than creating this content on-demand (which could impact performance), we implemented a pre-generation pipeline that sanitizes the content to prevent any XSS vulnerabilities, and stores it in R2 for faster retrieval.”
kandoh@reddthat.com 4 weeks ago
Yeah but you also add in the energy consumption of the data scrappers
digdilem@lemmy.ml 4 weeks ago
Surprised at the level of negativity here. Having had my sites repeatedly DDOSed offline by Claudebot and others scraping the same damned thing over and over again, thousands of times a second, I welcome any measures to help.
AWittyUsername@lemmy.world 4 weeks ago
I think the negativity is around the unfortunate fact that solutions like this shouldn’t be necessary.
dan@upvote.au 4 weeks ago
thousands of times a second
Modify your Nginx (or whatever web server you use) config to rate limit requests to dynamic pages, and cache them. For Nginx, you’d use either fastcgi_cache or proxy_cache depending on how the site is configured. Even if the pages change a lot, a cache with a short TTL (say 1 minute) can still help reduce load quite a bit while not letting them get too outdated.
Static content (and cached content) shouldn’t cause issues even if requested thousands of times per second. Following best practices like pre-compressing content using gzip, Brotli, and zstd helps a lot, too :)
Of course, this advice is just for “unintentional” DDoS attacks, not intentionally malicious ones. Those are often much larger and need different protection - often some protection on the network or load balancer before it even hits the server.
AnthropomorphicCat@lemmy.world 4 weeks ago
So the world is now wasting energy and resources to generate AI content in order to combat AI crawlers, by making them waste more energy and resources. Great! 👍
brucethemoose@lemmy.world 4 weeks ago
The energy cost of inference is overstated. Small models, or “sparse” models like Deepseek are not that expensive to run. Training is a one-time cost that still pales in comparison to industrial processes.
Basically, only Altman wants it to be cost prohibitive so he can have a monopoly. Also, he’s full of shit.
umbraroze@lemmy.world 4 weeks ago
I have no idea why the makers of LLM crawlers think it’s a good idea to ignore bot rules. The rules are there for a reason and the reasons are often more complex than “well, we just don’t want you to do that”. They’re usually more like “why would you even do that?”
Ultimately you have to trust what the site owners say. The reason why, say, your favourite search engine returns the relevant Wikipedia pages and not bazillion random old page revisions from ages ago is that Wikipedia said “please crawl the most recent versions using canonical page names, and do not follow the links to the technical pages (including history)”. Again: Why would anyone index those?
phoenixz@lemmy.ca 4 weeks ago
Because you are coming from the perspective of a reasonable person
These people are billionaires who expect to get everything for free. Rules are for the plebs, just take it already
pup_atlas@pawb.social 3 weeks ago
That’s what they are saying though. These shouldn’t be thought of as “rules”, they are suggestions near universally designed to point you to the most relevant content. Ignoring them isn’t “stealing something not meant to be captured”, it’s wasting time and resources of your own infra on something very likely to be useless to you.
T156@lemmy.world 4 weeks ago
Because it takes work to obey the rules, and you get less data for it. The theoretical comoetutor could get more ignoring those and get some vague advantage for it.
I’d not be surprised if the crawlers they used were bare-basic utilities set up to just grab everything without worrying about rule and the like.
EddoWagt@feddit.nl 4 weeks ago
They want everything, does it exist, but it’s not in their dataset? Then they want it.
They want their ai to answer any question you could possibly ask it. Filtering out what is and isn’t useful doesn’t achieve that
quack@lemmy.zip 4 weeks ago
Generating content with AI to throw off crawlers. I dread to think of the resources we’re wasting on this utter insanity now.
TorJansen@sh.itjust.works 4 weeks ago
And soon, the already AI-flooded net will be filled with so much nonsense that it becomes impossible for anyone to get some real work done. Sigh.
cantstopthesignal@sh.itjust.works 4 weeks ago
Some of us are only here to crank hog.
gac11@lemmy.world 4 weeks ago
AROOO!
biofaust@lemmy.world 4 weeks ago
I guess this is what the first iteration of the Blackwall looks like.
owl@infosec.pub 4 weeks ago
Gotta say “AI Labyrinth” sounds almost as cool.
surph_ninja@lemmy.world 4 weeks ago
I’m imagining a sci-fi spin on this where AI generators are used to keep AI crawlers in a loop, and they accidentally end up creating some unique AI culture or relationship in the process.
gmtom@lemmy.world 4 weeks ago
“I used the AI to destroy the AI”
Fluke@lemm.ee 4 weeks ago
And consumed the power output of a medium country to do it.
Yeah, great job! 👍
LeninOnAPrayer@lemm.ee 4 weeks ago
We truly are getting dumber as a species. We’re facing climate change but running some of the most power hungry processers in the world to spit out cooking recipes and homework answers for millions of people.
cantstopthesignal@sh.itjust.works 4 weeks ago
We had to kill the internet, to save the internet.
Asfalttikyntaja@sopuli.xyz 4 weeks ago
We have to kill the Internet, to save humanity.
drmoose@lemmy.world 4 weeks ago
Considering how many false positives Cloudflare serves i see nothing but misery coming from this.
Dave@lemmy.nz 4 weeks ago
In terms of Lemmy instances, if your instance is behind cloudflare and you turn on AI protection, federation breaks. So their tools are not very helpful for fighting the AI scraping.
Appoxo@lemmy.dbzer0.com 4 weeks ago
Can’t you configure exceptions for behaviours?
Xella@lemmy.world 4 weeks ago
Lol I work in healthcare and Cloudflare regularly blocks incoming electronic orders because the clinical notes “resemble” SQL injection. Nurses type all sorts of random stuff in their notes so there’s no managing that. Drives me insane!
Empricorn@feddit.nl 4 weeks ago
So we’re burning fossil fuels and destroying the planet so bots can try to deceive one another on the Internet in pursuit of our personal data. I feel like dystopian cyberpunk predictions didn’t fully understand how fucking stupid we are…
Flagstaff@programming.dev 4 weeks ago
They probably knew, but the truth is just boring and it’s funner to dramatize things, haha.
XeroxCool@lemmy.world 4 weeks ago
Will this further fuck up the inaccurate nature of AI results? While I’m rooting against shitty AI usage, the general population is still trusting it and making results worse will, most likely, make people believe even more wrong stuff.
ladel@feddit.uk 4 weeks ago
The article says it’s not poisoning the AI data, only providing valid facts. The scraper still gets content, just not the content it was aiming for.
melpomenesclevage@lemmy.dbzer0.com 4 weeks ago
and the data for the LLM is now salted with procedural garbage. it’s great!
ObsidianZed@lemmy.world 4 weeks ago
Until the AI generating the content starts hallucinating.
melpomenesclevage@lemmy.dbzer0.com 4 weeks ago
if you’re dumb enough to trust a large language model because someone told you “iTs Ai!” no amount of facts will be of great utility to you.
XeroxCool@lemmy.world 4 weeks ago
Thank you for catching that. Even reading through again, I couldn’t find it while skimming. With the mention of X2 and RSS, I assumed that paragraph would just be more technical description outside my knowledge. Instead, what I did hone in on was
“No real human would go four links deep into a maze of AI-generated nonsense.”
Leading me to be pessimistic.
melpomenesclevage@lemmy.dbzer0.com 4 weeks ago
If you’re dumb enough and care little enough about the truth, I’m not really going to try coming at you with rationality and sense. I’m down to do an accelerationism here. fuck it. burn it down.
remember; these companies all run at a loss. if we can hold them off for a while, they’ll stop getting so much investment.
einlander@lemmy.world 4 weeks ago
The problem I see with poisoning the data is the AI’s being trained for law enforcement hallucinating false facts used to arrest and convict people.
weremacaque@lemmy.world 4 weeks ago
You have Thirteen hours in which to solve this labyrinth before your baby AI becomes one of us, forever.
cantstopthesignal@sh.itjust.works 4 weeks ago
While AI David Bowie sings you rock lullabies.
missandry351@lemmings.world 4 weeks ago
This is getting ridiculous. Can someone please ban AI? Or at least regulate it somehow?
Slaxis@discuss.tchncs.de 4 weeks ago
The problem is, how? I can set it up on my own computer using open source models and some of my own code. It’s really rough to regulate that.
petaqui@lemmings.world 4 weeks ago
As for everything, it has good things, and bad things. We need to be careful and use it in a proper way, and the same thing applies to the ones creating this technology
VeloRama@feddit.org 4 weeks ago
Should have called it “Black ICE”.
lily33@lemm.ee 4 weeks ago
while allowing legitimate users and verified crawlers to browse normally.
What is a “verified crawler” though? What I worry about is, is it only big companies like Google that are allowed to have them now?
Onsotumenh@discuss.tchncs.de 4 weeks ago
Why do I have the feeling that I will end up in that nightmare with my privacy focused and ad-free Browser setup. I already end up in captcha hell too often because of it.
Revan343@lemmy.ca 4 weeks ago
Damned
ArasakaCloudflare ice walls are such a painbaltakatei@sopuli.xyz 4 weeks ago
Relevant excerpt from part 11 of Anathem (2008) by Neal Stephenson:
Artificial Inanity
Note: Reticulum=Internet, syndev=computer, crap~=spam “Early in the Reticulum—thousands of years ago—it became almost useless because it was cluttered with faulty, obsolete, or downright misleading information,” Sammann said. “Crap, you once called it,” I reminded him. “Yes—a technical term. So crap filtering became important. Businesses were built around it. Some of those businesses came up with a clever plan to make more money: they poisoned the well. They began to put crap on the Reticulum deliberately, forcing people to use their products to filter that crap back out. They created syndevs whose sole purpose was to spew crap into the Reticulum. But it had to be good crap.” “What is good crap?” Arsibalt asked in a politely incredulous tone. “Well, bad crap would be an unformatted document consisting of random letters. Good crap would be a beautifully typeset, well-written document that contained a hundred correct, verifiable sentences and one that was subtly false. It’s a lot harder to generate good crap. At first they had to hire humans to churn it out. They mostly did it by taking legitimate documents and inserting errors—swapping one name for another, say. But it didn’t really take off until the military got interested.” “As a tactic for planting misinformation in the enemy’s reticules, you mean,” Osa said. “This I know about. You are referring to the Artificial Inanity programs of the mid–First Millennium A.R.” “Exactly!” Sammann said. “Artificial Inanity systems of enormous sophistication and power were built for exactly the purpose Fraa Osa has mentioned. In no time at all, the praxis leaked to the commercial sector and spread to the Rampant Orphan Botnet Ecologies. Never mind. The point is that there was a sort of Dark Age on the Reticulum that lasted until my Ita forerunners were able to bring matters in hand.” “So, are Artificial Inanity systems still active in the Rampant Orphan Botnet Ecologies?” asked Arsibalt, utterly fascinated. “The ROBE evolved into something totally different early in the Second Millennium,” Sammann said dismissively. “What did it evolve into?” Jesry asked. “No one is sure,” Sammann said. “We only get hints when it finds ways to physically instantiate itself, which, fortunately, does not happen that often. But we digress. The functionality of Artificial Inanity still exists. You might say that those Ita who brought the Ret out of the Dark Age could only defeat it by co-opting it. So, to make a long story short, for every legitimate document floating around on the Reticulum, there are hundreds or thousands of bogus versions—bogons, as we call them.” “The only way to preserve the integrity of the defenses is to subject them to unceasing assault,” Osa said, and any idiot could guess he was quoting some old Vale aphorism. “Yes,” Sammann said, “and it works so well that, most of the time, the users of the Reticulum don’t know it’s there. Just as you are not aware of the millions of germs trying and failing to attack your body every moment of every day. However, the recent events, and the stresses posed by the Antiswarm, appear to have introduced the low-level bug that I spoke of.” “So the practical consequence for us,” Lio said, “is that—?” “Our cells on the ground may be having difficulty distinguishing between legitimate messages and bogons. And some of the messages that flash up on our screens may be bogons as well.”
finitebanjo@lemmy.world 4 weeks ago
Cloudflare kind of real for this. I love it.
It makes perfect sense for them as a business, infinite automated traffic equals infinite costs and lower server stability, but at the same time how often do giant tech companies do things that make sense these days?
MTK@lemmy.world 4 weeks ago
I swear someone released this exact thing a few weeks ago
Randomgal@lemmy.ca 4 weeks ago
I’m glad we’re burning the forests even faster in the name of identity politics.
perviouslyiner@lemmy.world 4 weeks ago
DNA Lounge has something similar - I think they even mentioned infinite JavaScript loops, and images that expand like zip-bombs.
jagermo@feddit.org 4 weeks ago
I am not happy with how much internet relies on cloudflare. However, they have a strong set of products
Deebster@infosec.pub 4 weeks ago
So they rewrote Nepenthes (or Iocaine, Spigot, Django-llm-poison, Quixotic, Konterfai, Caddy-defender, plus inevitably some Rust versions)
4am@lemm.ee 4 weeks ago
Imagine how much power is wasted on this unfortunate necessity.
Now imagine how much power will be wasted circumventing it.
Fucking clown world we live in
Demdaru@lemmy.world 4 weeks ago
On on hand, yes. On the other…imagine frustration of management of companies making and selling AI services. This is such a sweet thing to imagine.
halfapage@lemmy.world 4 weeks ago
Melvin_Ferd@lemmy.world 4 weeks ago
I just want to keep using uncensored AI that answers my questions. Why is this a good thing?
tfm@europe.pub 4 weeks ago
!TooDumbToImagine@europe.pub
zovits@lemmy.world 4 weeks ago
From the article it seems like they don’t generate a new labyrinth for every single time: Rather than creating this content on-demand (which could impact performance), we implemented a pre-generation pipeline that sanitizes the content to prevent any XSS vulnerabilities, and stores it in R2 for faster retrieval."