Comment on FediDB has stoped crawling until they get robots.txt support

• hendrik@palaver.p3x.de ⁨2⁩ ⁨months⁩ ago

  True. Question here is, if you run a federated service... Is that enough to assume you consent to federation?

  source

  • WhoLooksHere@lemmy.world ⁨2⁩ ⁨months⁩ ago

    Why invent implied consent when complicit consent has been the standard in robots.txt for ages now?

    source

    • hendrik@palaver.p3x.de ⁨2⁩ ⁨months⁩ ago

      I guess because it's in the specification? Or absent from it? But I'm not sure. Reading the ActivityPub specification is complicated, because you also need to read ActivityStreams and lots of other referen es. And I frequently miss stuff that is somehow in there.

      But generally we aren't Reddit where someone just says, no we prohibit third party use and everyone needs to use our app by our standards. The whole point of the Fediverse and ActivityPub is to interconnect. And to connect people across platforms. And it doen't even make lots of assumptions. The developers aren't forced to implement a Facebook clone. Or do something like Mastodon or GoToSocial does. They're relatively free to come up with new ideas and adopt things to their liking and use-cases. That's what makes us great and diverse.

      I -personally- see a public API endpoint as an invitation to use it. And that's kind of opposed to the consent thing.

      But with that said... We need some consensus in some areas. There are use cases where things arent obvious from the start. I'm just sad that everyone is ao agitated and seems to just escalate. I'm not sure if they tried talking to each other nicely. I suppose it's not a big deal to just implement the robots.txt and everyone can be happy. Without it needing some drama to get there.

      source

      • WhoLooksHere@lemmy.world ⁨2⁩ ⁨months⁩ ago

        Robots.txt started I’m 1994.

        It’s been a consensus for decades.

        Why throw it out and replace it with imied consent to scrape?

        That’s why I said legally there’s nothing they can do. If people want to scrape it they can and will.

        This is strictly about consent. Just because you can doesn’t mean you should yes?

        source

        • -> View More Comments
      • jmcs@discuss.tchncs.de ⁨2⁩ ⁨months⁩ ago

        You can consent to a federation interface without consenting to having a bot crawl all your endpoints.

        Just because something is available on the internet it doesn’t mean all uses are legitimate - this is effectively the same problem as AI training with stolen content.

        source

        • -> View More Comments
  • JustAnotherKay@lemmy.world ⁨2⁩ ⁨months⁩ ago

    if you run a federated services… Is that enough to assume you consent

    If she says yes to the marriage that doesn’t mean she permanently says yes to sex. I can run a fully air gapped “federated” instance if I want to

    source

    • hendrik@palaver.p3x.de ⁨2⁩ ⁨months⁩ ago

      Hmmh, I don't think we'll come to an agreement here. I think marriage is a good example, since that comes with lots of implicit consent. First of all you expect to move in together after you got engaged. You do small things like expect to eat dinner together. It's not a question anymore whether everyone cooks their own meal each day. And it extends to big things. Most people expect one party cares for the other once they're old. And stuff like that. And yeah. Intimacy isn't granted. There is a protocol to it. But I'm way more comfortable to make the moves on my partner, than for example place my hands on a stranger on the bus, and see if they take my invitation...

      Isn't that ho it works? I mean going with your analogy... Sure, you can marry someone and never touch each other or move in together. But that's kind of a weird one, in my opinion. Of course you should be able to do that. But it might require some more explicit agreement than going the default route. And I think that's what happened here. Assumptions have been made, those turned out to be wrong and now people need to find a way to deal with it so everyone's needs are met...

      source

      • JustAnotherKay@lemmy.world ⁨2⁩ ⁨months⁩ ago

        Going by your example

        Air gapping my service is the agreement you’re talking about in this analogy, but otherwise I do actually agree with you. There is a lot of implied consent, but I think we have a near miss misunderstanding on one part.

        In this scenario (analogies are nice but let’s get to reality) crawling the website to check the MAU, as harmless as it is, is still adding load to the server. A tiny amount, sure, but if you’re going to increase my workload by even 1% I wanna know beforehand. Thus, I put things on my website that say “don’t increase my workload” like robots.txt and whatnot.

        Other people aren’t this concerned with their workload, in which case it might be fine to go with implied consent. However, it’s always best to follow the best practices and just make sure with the owner of a server that it’s okay to do anything to their server IMO

        source

        • -> View More Comments