Open until your server is down because LLM are overloading it
Comment on This website is for humans
drmoose@lemmy.world 1 day ago
As someone who’s been on the web since the 90s I hate this.
The web was designed to be user agent agnostic. Desktop, phone, fridge, ai agents, curl, python script - whatever agent you are using shouldn’t matter for access. That’s the whole point of open internet, period.
whyNotSquirrel@sh.itjust.works 1 day ago
kescusay@lemmy.world 1 day ago
At my company, we had to implement all sorts of WAF rules precisely for that reason. Those things are fucking aggressive.
bravesilvernest@lemmy.ml 1 day ago
Same. And just because page size is “low” doesn’t mean shit when they’re flooding requests. Try having public research data and watch how much your costs go up just due to load balancer throughput.
drmoose@lemmy.world 1 day ago
overloading from 200kb of html? We’re not in dialup era anymore
xthexder@l.sw0.com 15 hours ago
It’s one request Michael, what could it cost? 200kb?
drmoose@lemmy.world 9 hours ago
Literally free. So many tech illiterate people in this threat that still think website hosting costs money.
JustARaccoon@lemmy.world 1 day ago
They did have a lot of concerns with abuse though and you can see that in the way the cookies debate went before they were supported in their current form. I think AI crawlers tanking bandwidths for websites and misusing the data they scrape would 100% be something the Mozilla from back then would’ve had concerns over allowing or encouraging.
drmoose@lemmy.world 1 day ago
You’re conflating two different issues. The topic is “for whom the web is for?” not banwidth distribution and optimization.
If LLM bot is being abusive then that’s no different from any other user agent behaving like this and we should expand these protections from intentional/unintentional ddos irrelevant of user agent.
ernest314@lemmy.zip 1 day ago
I think your starting point (allowing bot user agents to crawl the web has overlooked benefits) is a good one, but things aren’t black and white–there are clear drawbacks, too. Bots obviously have an orders of magnitude higher potential for abuse; to the point where bot traffic–as it currently stands in the real world–is qualitatively different from human traffic.
we should expand these protections from intentional/unintentional ddos irrelevant of user agent.
Sure, but targeted regulation based on heuristics (in this case, user agent) is also a widely accepted practice. DUI laws exist, even though the goal (fewer murders and safer roads) are already separately regulated.
Would it be nice if we didn’t have to do this? Or there were some other solution? Sure, but I have no idea where to even start, unfortunately.
frezik@lemmy.blahaj.zone 23 hours ago
Instructions unclear, built whole site with nested tables.
caseyweederman@lemmy.ca 8 hours ago
Each one had better be in its own iframe.
Allero@lemmy.today 1 day ago
When the Web was first designed, some of the concerns we have today were nonexistent.
I believe in freedom of information, and would love for the information I share to be accessed in any way a given user wants.
But I have to stand defensive and support the author here, too. The modern LLM boom aims to essentially replace original resources with AI-generated summaries step by step. This is detrimental to the Internet, and to knowledge as we know and preserve it.
First, there is an event commonly called Google Zero, which is briefly mentioned in the article. If you don’t know what it is, it is the not-so-hypothetical-anymore moment when Google (or, really, any other large player) essentially accumulates all information on the Web, feeds it to AI, and since then doesn’t serve links anymore, going straight to answers based on training data. Users will jump to this - they already do - because it offers convenience. But for any independent creators it means having no audience, no money, and no means to produce new quality content, trapping users in a self-containing loop that loses nuance, actuality, and truthfulness. This goes beyond cooking recipes and personal notes - it permeates science, political discussion, and much more.
Second, LLMs multiply traffic coming to sites, which becomes an infrastructural problem. Bots access sites at much higher rates than humans do, and when their intent is to scrape your entire website every now and again and there are dozens of them, this becomes huge.
Third, having proprietary models train on the data I provide without any attribution, copyright etc. makes giant corporation profit off my back, while at the same thing making it so that less genuine users will see what I produce. This means careers of authors, journalists etc. are dying, and this also mean they abuse each and every one of us without any consent.
Fourth, and I wonder if you see it by now, LLMs and the way they represent data, along with SEO tools meant to drive information through the search bots, begin to shape how we talk. All I say doesn’t have to be a list of points, yet it is. It could be less verbose, more readable, yet it is the way it is. Because when we interact with the products of such developments too much, we begin shaping our own language in a way that is less human-readable and more meant for machines, without us knowing. This is a real issue of communication.
So, as much as I hate it, I’m gonna protect a lot of the data I share.
drmoose@lemmy.world 1 day ago
I fundamentally disagree with all of your claims. The web was already ruined by SEO farms and I know that cause I worked in organic growth for years, AI is not making this worse.
The traffic argument is non-sensical in 2025. serving 200kb html file costs literally nothing. So if you want to write and share something you can do it without spending a single penny, ever.
I understand the frustration and confusion here but all of this whining is lacking any real vision. Information should be free and accessible to all and the rest can be solved without changing this core principle.
So while you “protect” what you share we all will continue to grow and share information freely and actualy contribute real change not start breaking the looms.
Allero@lemmy.today 1 day ago
Locking information into corporate-controlled loops is antithetical to freedom and accessibility.
Having singular proprietary point of entry, or even few of them, into the entire knowledge of mankind is not sharing.
This is the part people are willing to protect. Actual peer-to-peer sharing of information, with as little private choke points as possible.