if you run a federated services… Is that enough to assume you consent
If she says yes to the marriage that doesn’t mean she permanently says yes to sex. I can run a fully air gapped “federated” instance if I want to
Comment on FediDB has stoped crawling until they get robots.txt support
hendrik@palaver.p3x.de 2 weeks agoTrue. Question here is, if you run a federated service... Is that enough to assume you consent to federation?
JustAnotherKay@lemmy.world 2 weeks ago
hendrik@palaver.p3x.de 2 weeks ago
Hmmh, I don't think we'll come to an agreement here. I think marriage is a good example, since that comes with lots of implicit consent. First of all you expect to move in together after you got engaged. You do small things like expect to eat dinner together. It's not a question anymore whether everyone cooks their own meal each day. And it extends to big things. Most people expect one party cares for the other once they're old. And stuff like that. And yeah. Intimacy isn't granted. There is a protocol to it. But I'm way more comfortable to make the moves on my partner, than for example place my hands on a stranger on the bus, and see if they take my invitation...
Isn't that ho it works? I mean going with your analogy... Sure, you can marry someone and never touch each other or move in together. But that's kind of a weird one, in my opinion. Of course you should be able to do that. But it might require some more explicit agreement than going the default route. And I think that's what happened here. Assumptions have been made, those turned out to be wrong and now people need to find a way to deal with it so everyone's needs are met...
JustAnotherKay@lemmy.world 2 weeks ago
Going by your example
Air gapping my service is the agreement you’re talking about in this analogy, but otherwise I do actually agree with you. There is a lot of implied consent, but I think we have a near miss misunderstanding on one part.
In this scenario (analogies are nice but let’s get to reality) crawling the website to check the MAU, as harmless as it is, is still adding load to the server. A tiny amount, sure, but if you’re going to increase my workload by even 1% I wanna know beforehand. Thus, I put things on my website that say “don’t increase my workload” like robots.txt and whatnot.
Other people aren’t this concerned with their workload, in which case it might be fine to go with implied consent. However, it’s always best to follow the best practices and just make sure with the owner of a server that it’s okay to do anything to their server IMO
hendrik@palaver.p3x.de 2 weeks ago
I don't think that'll work. Asking for consent and retrieving the robots.txt is yet another request with a similar workload. So by that logic, we can't do anything on the internet. Since asking for consent is work and that requires consent, which requires consent... And if you're concerned with efficiency alone, cut the additional asking and complexity by just straightforward doing the single request.
Plus, it's not even that complex. Sending a few bytes of JSON with daily precalculated numbers is a fraction of what a single user interaction does. It's maybe zero point something of a request. Or with a lots of more zero's in-between if we look at what a server does each day. I mean every single refresh of the website or me opening the app loads several files, API endpoints, regularly loads hundreds of kilobytes of Javascript, images etc. There are lots of calculations and database requests involved to display several posts along with votes etc. I'd say one single pageview of me counts like the FediDB collecting sttats each day for like 1000 years.
I invented these numbers. They're wrong. But I think you get what I'm trying to say... For all practical purposes, these requests are for free and have zero cost. Plus if it's efficientcy, it's always a good idea not to ask to ask, but outright do it and deal with it while answering. So it rally can't be computational cost. It has to be consent.
notfromhere@lemmy.ml 2 weeks ago
How is it air gapped and federated? Do you unairgap it periodically for a refresh then reairgap it? I’ve not heard of airgapped federated servers before and am intrigued. Is it purely for security purposes or also bandwidth savings? Are there other reasons one may want to run an air gapped instance?
WhoLooksHere@lemmy.world 2 weeks ago
Why invent implied consent when complicit consent has been the standard in robots.txt for ages now?
hendrik@palaver.p3x.de 2 weeks ago
I guess because it's in the specification? Or absent from it? But I'm not sure. Reading the ActivityPub specification is complicated, because you also need to read ActivityStreams and lots of other referen es. And I frequently miss stuff that is somehow in there.
But generally we aren't Reddit where someone just says, no we prohibit third party use and everyone needs to use our app by our standards. The whole point of the Fediverse and ActivityPub is to interconnect. And to connect people across platforms. And it doen't even make lots of assumptions. The developers aren't forced to implement a Facebook clone. Or do something like Mastodon or GoToSocial does. They're relatively free to come up with new ideas and adopt things to their liking and use-cases. That's what makes us great and diverse.
I -personally- see a public API endpoint as an invitation to use it. And that's kind of opposed to the consent thing.
But with that said... We need some consensus in some areas. There are use cases where things arent obvious from the start. I'm just sad that everyone is ao agitated and seems to just escalate. I'm not sure if they tried talking to each other nicely. I suppose it's not a big deal to just implement the robots.txt and everyone can be happy. Without it needing some drama to get there.
WhoLooksHere@lemmy.world 2 weeks ago
Robots.txt started I’m 1994.
It’s been a consensus for decades.
Why throw it out and replace it with imied consent to scrape?
That’s why I said legally there’s nothing they can do. If people want to scrape it they can and will.
This is strictly about consent. Just because you can doesn’t mean you should yes?
rimu@piefed.social 2 weeks ago
Let's see about that.
Wikipedia lists http://www.robotstxt.org as the official homepage of robots.txt and the "Robots Exclusion Protocol". In the FAQ at http://www.robotstxt.org/faq.html the first entry is "What is a WWW robot?" http://www.robotstxt.org/faq/what.html. It says:
That's not FediDB. That's not even nodeinfo.
hendrik@palaver.p3x.de 2 weeks ago
I just think you're making it way more simple than it is... Why not implement 20 other standards that have been around for 30 years? Why not make software perfect and without issues? Why not anticipate what other people will do with your public API endpoints in the future?
There could be many reasons. They forgot, they didn't bother, they didn't consider themselves to be the same as a commercial Google or Yandex crawler... That's why I keep pushing for information and refuse to give a simple answer. Could be an honest mistake. Could be honest and correct to do it and the other side is wrong, since it's not a crawler alike Google or the AI copyright thieves... Could be done maliciously. In my opinion, it's likely that it hadn't been an issue before, the situation changed and now this needs a solution. And we're getting one. Seems at least FediDB took it offline and they're working on robots.txt support. They did not refuse to do it. So it's fine. And I can't comment on why it hadn't been in place. I'm not involved with that project and the history of it's development.
jmcs@discuss.tchncs.de 2 weeks ago
You can consent to a federation interface without consenting to having a bot crawl all your endpoints.
Just because something is available on the internet it doesn’t mean all uses are legitimate - this is effectively the same problem as AI training with stolen content.
hendrik@palaver.p3x.de 2 weeks ago
Yes. I wholeheartedly agree. Not every use is legitimate. But I'd really need to know what exactly happeded and the whole story to judge here. I'd say if it were a proper crawler, they'd need to read the robots.txt. That's accepted consensus. But is that what's happened here?
And I mean the whole thing with consensus and arbitrary use cases is just complicated. I have a website, and a Fediverse instance. Now you visit it. Is this legitimate? We'd need to factor in why I put it there. And what you're doing with that information. If it's my blog, it's obviously there for you to read it... Or is it!? But that's implied consent. I'd argue this is how the internet works. And most of the times it's super easy to tell what's right an what is wrong. But sometimes it isn't.