Robots.txt isn’t even a rule, it’s a request.
“Please do not ask for the following content if you are a robot”.
If you don’t want someone to look at your content, you ultimately have to not give it to them, not just ask them to not ask.
Comment on OpenAI and Anthropic are ignoring an established rule that prevents bots scraping online content
treefrog@lemm.ee 3 months ago
A spokeswoman for OpenAI declined to comment beyond pointing BI to a corporate blogpost from May, in which the company says it takes web crawler permissions “into account each time we train a new model.
The translation for this is do we stand to profit more than we stand to be fined
ricecake@sh.itjust.works 3 months ago
Grimy@lemmy.world 3 months ago
They stand to profit if this is made into a real law.
Any regulation on AI just kill off their competition at this point. They are both lobbying for it and numerous proposed “anti-AI” laws have been their doing.
cbarrick@lemmy.world 3 months ago
They can’t even be punished.
robots.txtis just a convention, not a regulation. It’s totally not enforceable.The only legal framework we have is copyright law. Those who oppose this behavior will have to demonstrate copyright violation, and that may be difficult to do since the law hasn’t caught up.
lemmyvore@feddit.nl 3 months ago
It’s true robots is not regulation but if it’s proven they ignore it on purpose it will be a major point in future lawsuits. And those are the next step.
conciselyverbose@sh.itjust.works 3 months ago
It won’t have any relevance at all.
Either scraping to transform the information in the page is fair use, and consent isn’t necessary, or it is not fair use, and the absence of a robots.txt doesn’t constitute consent. There’s no middle ground where a robots.txt can mean anything.
treefrog@lemm.ee 3 months ago
Yeah I know. But I wanted to point out that the comment in the article wasn’t so much a real consideration as business risk of analysis 101. Along with a healthy amount of corporate spin.