I swear someone released this exact thing a few weeks ago
Cloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.
Submitted 1 year ago by Tea@programming.dev to technology@lemmy.world
https://blog.cloudflare.com/ai-labyrinth/
Comments
AnthropomorphicCat@lemmy.world 1 year ago
So the world is now wasting energy and resources to generate AI content in order to combat AI crawlers, by making them waste more energy and resources. Great! 👍
brucethemoose@lemmy.world 1 year ago
The energy cost of inference is overstated. Small models, or “sparse” models like Deepseek are not that expensive to run. Training is a one-time cost that still pales in comparison to industrial processes.
Basically, only Altman wants it to be cost prohibitive so he can have a monopoly. Also, he’s full of shit.
perviouslyiner@lemmy.world 1 year ago
DNA Lounge has something similar - I think they even mentioned infinite JavaScript loops, and images that expand like zip-bombs.
fubarx@lemmy.world 1 year ago
So this showed up last week: github.com/raminf/RoboNope-nginx
Similar vibe, minus the AI.
Empricorn@feddit.nl 1 year ago
So we’re burning fossil fuels and destroying the planet so bots can try to deceive one another on the Internet in pursuit of our personal data. I feel like dystopian cyberpunk predictions didn’t fully understand how fucking stupid we are…
Flagstaff@programming.dev 1 year ago
They probably knew, but the truth is just boring and it’s funner to dramatize things, haha.
jagermo@feddit.org 1 year ago
I am not happy with how much internet relies on cloudflare. However, they have a strong set of products
Deebster@infosec.pub 1 year ago
So they rewrote Nepenthes (or Iocaine, Spigot, Django-llm-poison, Quixotic, Konterfai, Caddy-defender, plus inevitably some Rust versions)
RelativeArea1@sh.itjust.works 1 year ago
this is some fucking stupid situation, we somewhat got a faster internet and these bots messing each other are hugging the bandwidth.
drmoose@lemmy.world 1 year ago
Lol website traffic accounts for like 1% of bandwidth budget. 1 netflix movie is like 20k web pages.
dual_sport_dork@lemmy.world 1 year ago
Especially since the solution I cooked up for my site was to identify the incoming requests from these damn bots – which is not difficult, since they ignore all directives and sanity and try to slam your site with like 200+ requests per second, that makes 'em easy to spot – and simply IP ban them.
In fact, anybody who doesn’t exhibit a sane crawl rate gets blocked from my site automatically. For a while, most of them were coming from Russian IP address zones for some reason. These days Amazon is the worst offender, I guess their Rufus AI or whatever the fuck it is tries to pester other retail sites to “learn” about products rather than sticking to its own domain.
Fuck 'em. Route those motherfuckers right to /dev/null.
desktop_user@lemmy.blahaj.zone 1 year ago
the only problem with that solution being applied to generic websites is schools and institutions can have many legitimate users from one IP address and many sites don’t want a chance to accidentally block one.
morrowind@lemmy.ml 1 year ago
Cloudflare offers that too, but you can’t always tell
Buelldozer@lemmy.today 1 year ago
and try to slam your site with like 200+ requests per second
Your solution would do nothing to stop the crawlers that are operating 10ish rps. There’s ones out there operating at a mere 2rps but when multiple companies are doing it at the same time 24x7x365 it adds up.
Some incredibly talented people have been battling this since last year and your solution has been tried multiple times. It’s not effective in all instances and can require a LOT of manual intervention and SysAdmin time.
ininewcrow@lemmy.ca 1 year ago
It’s what I’ve been saying about technology for the past decade or two … we’ve hit an upper limit to our technological development … that limit is on individual human greed where small groups of people or massively wealthy people hinder or delay any further development because they’re always trying to find ways to make money off it, prevent others from making money off it, monopolize an area or section of society … capitalism is literally our world’s bottleneck and it’s being choked off by an oddly shaped gold bar at this point.
melpomenesclevage@lemmy.dbzer0.com 1 year ago
nothing can be improved while capitalism exists; all improvement will be seized and used to oppress.
QuarterSwede@lemmy.world 1 year ago
The problem you aren’t recognizing is that, until humans are no longer driven by self preservation, there will always be oppression in any system. They all have and will continue to breakdown. It’s easy to blame capitalism but even socialist systems eventually cave under the weight of greed and power. We are the problem mon frère.
morrowind@lemmy.ml 1 year ago
How can authority not exist? That’s staggeringly broad
IrateAnteater@sh.itjust.works 1 year ago
That’s not really relevant here. This is more of a “genie is out of the bottle and now we have to learn how to deal with it situation”. The idea and technology of bots and AI training already exists. There’s no socioeconomic system that is going to magically make that go away.
lily33@lemm.ee 1 year ago
while allowing legitimate users and verified crawlers to browse normally.
What is a “verified crawler” though? What I worry about is, is it only big companies like Google that are allowed to have them now?
fuckwit_mcbumcrumble@lemmy.dbzer0.com 1 year ago
Cloudflare isn’t the best at blocking things. As long as your crawler isn’t horribly misconfigured you shouldn’t have much issues.
melpomenesclevage@lemmy.dbzer0.com 1 year ago
I dunno. I don’t find any sympathy with any of these fuckers though. this is not a generally useful technology, it is not something the average person ever needs to see, and honestly, just fuck em. Fuck anyone messing with open source to engorge the garbage dispenser.
lily33@lemm.ee 1 year ago
Any accessibility service will also see the “hidden links”, and while a blind person with a screen reader will notice if they wonder off into generated pages, it will waste their time too.
Also, I don’t know about you, but I absolutely have a use for crawling X, Google maps, Reddit, YouTube, and getting information from there without interacting with the service myself.
wingiee@lemm.ee 1 year ago
I assume a crawler which adheres to robots.txt
lily33@lemm.ee 1 year ago
I would love to think so. But the word “verified” suggests more.
XeroxCool@lemmy.world 1 year ago
Will this further fuck up the inaccurate nature of AI results? While I’m rooting against shitty AI usage, the general population is still trusting it and making results worse will, most likely, make people believe even more wrong stuff.
melpomenesclevage@lemmy.dbzer0.com 1 year ago
If you’re dumb enough and care little enough about the truth, I’m not really going to try coming at you with rationality and sense. I’m down to do an accelerationism here. fuck it. burn it down.
remember; these companies all run at a loss. if we can hold them off for a while, they’ll stop getting so much investment.
einlander@lemmy.world 1 year ago
The problem I see with poisoning the data is the AI’s being trained for law enforcement hallucinating false facts used to arrest and convict people.
ladel@feddit.uk 1 year ago
The article says it’s not poisoning the AI data, only providing valid facts. The scraper still gets content, just not the content it was aiming for.
XeroxCool@lemmy.world 1 year ago
Thank you for catching that. Even reading through again, I couldn’t find it while skimming. With the mention of X2 and RSS, I assumed that paragraph would just be more technical description outside my knowledge. Instead, what I did hone in on was
“No real human would go four links deep into a maze of AI-generated nonsense.”
Leading me to be pessimistic.
melpomenesclevage@lemmy.dbzer0.com 1 year ago
if you’re dumb enough to trust a large language model because someone told you “iTs Ai!” no amount of facts will be of great utility to you.
ObsidianZed@lemmy.world 1 year ago
Until the AI generating the content starts hallucinating.
melpomenesclevage@lemmy.dbzer0.com 1 year ago
and the data for the LLM is now salted with procedural garbage. it’s great!
AtomicHotSauce@lemmy.world 1 year ago
That’s just BattleBots with a different name.
supersquirrel@sopuli.xyz 1 year ago
No, it is far less environmentally friendly than warrior bots made of metal, plastic, and electronics full of nasty little things like batteries blasting, sawing, burning and smashing one another to pieces.
ininewcrow@lemmy.ca 1 year ago
They should program the actions and reactions of each system to actual battle bots and then televise the event for our entertainment.
singletona@lemmy.world 1 year ago
Then get bored when it devolves into a wedge meta.
aviationeast@lemmy.world 1 year ago
You’re not wrong.
IrateAnteater@sh.itjust.works 1 year ago
Ok, I now need a screensaver that I can tie to a cloudflare instance that visualizes the generated “maze” and a bot’s attempts to get out.
Revan343@lemmy.ca 1 year ago
Damned
ArasakaCloudflare ice walls are such a pain