Submitted 8 months ago by TravisKelce@lemmy.world to technology@lemmy.world
https://www.theverge.com/24067997/robots-txt-ai-text-file-web-crawlers-spiders
As unscrupulous AI companies crawl for more and more data, the basic social contract of the web is falling apart.
Honestly it seems like in all aspects of society the social contract is being ignored these days, that’s why things seem so much worse now.
It’s abuse, plain and simple.
Governments could do something about it, if they weren’t overwhelmed by bullshit from bullshit generators instead and lead by people driven by their personal wealth.
these days
When, at any point in history, have people acknowledged that there was no social change or disruption and everyone was happy?
Well the trump era has shown that ignoring social contracts and straight up crime are only met with profit and slavish devotion from a huge community of dipshits. So. Y’know.
Only if you’re already rich or in the right social circles though. Everyone else gets fined/jail time of course.
Meh maybe. I know plenty of people who get away with all kinds of crap without money or connections.
Alternative title: Capitalism doesn’t care about morals and contracts. It wants to make more money.
Exactly. Capitalism spits in the face of the concept of a social contract, especially if companies themselves didn’t write it.
Capitalism, at least, in a lassie-faire marketplace, operates on a social contract, fiat money is an example of this. The market decides, the people decide. Are there ways to amass a certain amount of money to make people turn blind eyes? For sure, but all systems have their ways to amass power, no matter what
Capitalism is a concept, it can’t care if it wanted and it even can’t want to begin with. It’s the humans. You will find greedy, immoral ones in every system and they will make it miserable for everyone else.
Capitalism is the widelly accepted self-serving justification of those people’s for their acts.
The real problem is in the “widelly accepted” part: a sociopath killing an old lady and justifying it because “she looked funny at me” wouldn’t be “widelly accepted” and Society would react in a suitable way, but if said sociopath scammed the old lady’s pension fund because (and this is a typical justification in Investment Banking) “the opportunity was there and if I didn’t do it somebody else would’ve, so better be me and get the profit”, it’s deemed “acceptable” and Society does not react in a suitable way.
The open and free web is long dead.
just thinking about robots.txt as a working solution to people that literally broker in peoples entire digital lives for hundreds of billions of dollars is so … quaint.
It’s up there with Do-Not-Track.
Completely pointless because it’s not enforced
Do-Not-Track, AKA, “I’ve made my browser fingerprint more unique for you, please sell my data”
I bet at least one site I’ve visited has enforced it
I would be shocked if any big corpo actually gave a shit about it.
if exists("/robots.txt"): no it fucking doesn't
Robots.txt is in theory meant to be there so that web crawlers don’t waste their time traversing a website in an inefficient way. It’s there to help, not hinder them. There is a social contract being broken here and in the long term it will have a negative impact on the web.
A social contract only holds when both sides stand to gain, so if one exists then it’s “you’re going to crawl the site anyway, so if you do it in the optimized manner described in this file we won’t take steps to hinder you.” It’s not, and never has been, a way to block bots from crawling your website content.
Yeah I always found it surprising that everyone just agreed to follow a text file on a website on how to act. It’s one of the worst thought out/significant issues with browsing still out there from the beginning pretty much.
We need laws mandating respect of robots.txt. This is what happens when you don’t codify stuff
It’s a bad solution to a problem anyway. If we are going to legally mandate a solution I want to take the opportunity to come up with an actually better fix than the hacky solution that is robots.txt
AI companies will probably get a free pass to ignore robots.txt even if it were enforced by law. That’s what they’re trying to do with copyright and it looks likely that they’ll get away with it.
Turning that into a law is ridiculous - you really can’t consider that more than advisory unless you enforce it with technical means. For example, maybe put it behind a login or captcha if you want only humans to see it
Are you aware of what “unlisted” means?
you can’t really make laws in the united states
The battle cry of conservatives everywhere: It’s too hard!
Except if it involves oppressing minorities and women. Then it’s a moral imperative worth all the time and money you can shovel at it regardless of whether the desired outcome is realistic or not.
Sounds like the type of thing that would either be unenforceable or profitable to violate compared to the fines.
I hope not laws tend to get outdated real fast. Who knows robots.txt might not even be used in the future and it just there adding space because of law reasons.
You can describe the law in a similar way to a specification, and you can make it as broad as needed. Something like the file name shouldn’t ever come up as an issue.
robots.txt is a 30 year old standard. If we can write common sense laws around things like email and VoIP, we can do it for web standards too.
We don’t need new laws we just need enforcement of existing laws. It is already illegal to copy copyrighted content, it’s just that the AI companies do it anyway and no one does anything about it.
Enforcing respect for robots.txt doesn’t matter because the AI companies are already breaking the law.
robots.txt has been an unofficial standard for 30 years and its augmented with sitemap.xml to help index uncrawlable pages, and Schema.org to expose contents for Semantic Web. I’m not stating it shouldn’t not be a law, but to suggest changing norms as a reason is a pretty weak counterargument, man.
Why? What would you like to achieve and how would that help?
All my scrapping scripts go to shit…please no, I need automation to live…
Most every other social contract has been violated already. If they don’t ignore robots.txt, what is left to violate?? Hmm??
It’s almost as if leaving things to social contracts vs regulating them is bad for the layperson… 🤔
Nah fuck it. The market will regulate itself! Tax is theft and I don’t want that raise or I’ll get in a higher tax bracket and make less!
This can actually be an issue for poor people, not because of tax bracketa but because of income-based assistance cutoffs. If $1/hr raise throws you above those cutoffs, that extra $160 could cost you $500 in food assistance, $5-$10/day for school lunch, or get you kicked out of government subsidied housing.
Yet another form of persecution that the poor actually suffer and the rich pretend to.
God the number of people I’ve heard say this over the years is nuts.
And then the companies hit the “trust thermocline”, customers leave them in droves and companies wonder how this could’ve happened.
I got it was sarcasm, but it’s always good to add a /s just in case
Yea, because authoritarianism is well known to be sooooo good for the layperson.
They didn’t violate the social contact, they disrupted it.
True innovation. So brave.
What social contract? When sites regularly have a robots.txt that says “only Google may crawl”, and are effectively helping enforce a monolopy, that’s not a social contract I’d ever agree to.
When sites regularly have a robots.txt that says “only Google may crawl”
Is that actually true?
If so, why would they do that?
No laws to govern so they can do anything they want. Blame boomer politicians not the companies.
Why not blame the companies ? After all they are the ones that are doing it, not the boomer politicians.
And in the long term they are the ones that risk to be “punished”, just imagine people getting tired of this shit and starting to block them at a firewall level…
Because the politicians also created the precedent that anything you can get away with, goes. They made the game, defined the objective, and then didn’t adapt quickly so that they and their friends would have a shot at cheating.
There is absolutely no narrative of “what can you do for your country” anymore. It’s been replaced by the mottos of “every man for himself” and “get while the getting’s good”.
I think that good behavior is implicitly mandated even if there’s nobody to punish you if you don’t.
Lol
TIL that robots.txt is a thing
hmm, i though websites just blocked crawler traffic directly? I know one site in particular has rules about it, and will even go so far as to ban you permanently if you continually ignore them.
Strong “the constitution is a piece of paper” energy right there
I explicitly have my robots.txt set to block out AI crawlers, but I don’t know if anyone else will observe the protocol. They should have tools I can submit a sitemap.xml against to know if i’ve been parsed. Until they bother to address this, I can only assume their intent is hostile and if anyone is serious about building a honeypot and exposing the tooling for us to deploy at large, my options are limited.
This is a very interesting read. It is very rarely people on the internet agree to follow 1 thing without being forced
🤣🤣🤣🤣🤣🤣🤣 “robots.txt is a social contract” 🤣🤣🤣🤣🤣🤣🤣 🤡
This is the best summary I could come up with:
If you hosted your website on your computer, as many people did, or on hastily constructed server software run through your home internet connection, all it took was a few robots overzealously downloading your pages for things to break and the phone bill to spike.
AI companies like OpenAI are crawling the web in order to train large language models that could once again fundamentally change the way we access and share information.
In the last year or so, the rise of AI products like ChatGPT, and the large language models underlying them, have made high-quality training data one of the internet’s most valuable commodities.
You might build a totally innocent one to crawl around and make sure all your on-page links still lead to other live pages; you might send a much sketchier one around the web harvesting every email address or phone number you can find.
The New York Times blocked GPTBot as well, months before launching a suit against OpenAI alleging that OpenAI’s models “were built by copying and using millions of The Times’s copyrighted news articles, in-depth investigations, opinion pieces, reviews, how-to guides, and more.” A study by Ben Welsh, the news applications editor at Reuters, found that 606 of 1,156 surveyed publishers had blocked GPTBot in their robots.txt file.
“We recognize that existing web publisher controls were developed before new AI and research use cases,” Google’s VP of trust Danielle Romain wrote last year.
The original article contains 2,912 words, the summary contains 239 words. Saved 92%. I’m a bot and I’m open source!
Also, by the way, violating a basic social contract to not work towards triggering an intelligence explosion that will likely replace all biological life on Earth with computronium, but who’s counting? :)
good. robots.txt was always a bad idea
Why should I care about a text file lol
sigh. Of course they are ...
Wow I’m shocked! Just like how OpenAI preached for “privacy and ethics” and went deafly silent on data hoarding and scraping, then privatizes their stolen scraped data. If they insist their data collection to be private, then it needs regular external audits by strict data privacy firms just like they do with security.
palordrolap@kbin.social 8 months ago
Put something in robots.txt that isn't supposed to be hit and is hard to hit by non-robots. Log and ban all IPs that hit it.
Imperfect, but can't think of a better solution.
lvxferre@mander.xyz 8 months ago
Good old honeytrap. I’m not sure, but I think that it’s doable.
Have a honeytrap page somewhere in your website. Make sure that legit users won’t access it. Disallow crawling those pages through robots.txt.
Then if some crawler still access it, you could simply record it in the logs and ban the relevant IPs. Or you could be really nasty, and fill the page with poison - nonsensical text that would look like something that humans would write.
CosmicTurtle@lemmy.world 8 months ago
I think I used to do something similar with email spam traps. Not sure if it’s still around but basically you could help build NaCL lists by posting an email address on your website somewhere that was visible in the source code but not visible to normal users, like in a div that was way on the left side of the screen.
Anyway, spammers that do regular expression searches for email addresses would email it and get their IPs added to naughty lists.
I’d love to see something similar with robots.
thefactremains@lemmy.world 8 months ago
Even better. Build a WordPress plugin to do this.
KairuByte@lemmy.dbzer0.com 8 months ago
I’m the idiot human that digs through robots.txt and the site map to see things that aren’t normally accessible by an end user.
lol@discuss.tchncs.de 8 months ago
Blackmist@feddit.uk 8 months ago
“Help, my website no longer shows up in Google!”
PM_Your_Nudes_Please@lemmy.world 8 months ago
Yeah, this is a pretty classic honeypot method. Basically make something available but inaccessible to the normal user. Then you know anyone who accesses it is not a normal user.
I’ve even seen this done with Steam achievements before; There was a hidden game achievement which was only available via hacking. So anyone who used hacks immediately outed themselves with a rare achievement that was visible on their profile.
Link@rentadrunk.org 8 months ago
That’s a bit annoying as it means you can’t 100% the game as there will always be one achievement you can’t get.
CileTheSane@lemmy.ca 8 months ago
There are tools that just flag you as having gotten an achievement on Steam, you don’t even have to have the game open to do it. I’d hardly call that ‘hacking’.
Aatube@kbin.social 8 months ago
robots.txt is purely textual; you can't run JavaScript or log anything. Plus, one who doesn't intend to follow robots.txt wouldn't query it.
BrianTheeBiscuiteer@lemmy.world 8 months ago
If it doesn’t get queried that’s the fault of the webscraper. You don’t need JS built into the robots.txt file either. Just add some line like:
here-there-be-dragons.htmlAny client that hits that page (and maybe doesn’t pass a captcha check) gets banned. Or even better, they get a long stream of nonsense.
ShitpostCentral@lemmy.world 8 months ago
You’re second point is a good one, but you absolutely can log the IP which requested robots.txt. That’s just a standard part of any http server ever, no JavaScript needed.
ricecake@sh.itjust.works 8 months ago
People not intending to follow it is the real reason not to bother, but it’s trivial to track who downloaded the file and then hit something they were asked not to.
Like, 10 minutes work to do right. You don’t need js to do it at all.
Ultraviolet@lemmy.world 8 months ago
Better yet, point the crawler to a massive text file of almost but not quite grammatically correct garbage to poison the model.
odelik@lemmy.today 8 months ago
Maybe one of the lorem ipsum generators could help.
nullPointer@programming.dev 8 months ago
a bad-bot .htaccess trap.