Comment on FediDB has stoped crawling until they get robots.txt support
hendrik@palaver.p3x.de 2 months agoHmmh, I don't think we'll come to an agreement here. I think marriage is a good example, since that comes with lots of implicit consent. First of all you expect to move in together after you got engaged. You do small things like expect to eat dinner together. It's not a question anymore whether everyone cooks their own meal each day. And it extends to big things. Most people expect one party cares for the other once they're old. And stuff like that. And yeah. Intimacy isn't granted. There is a protocol to it. But I'm way more comfortable to make the moves on my partner, than for example place my hands on a stranger on the bus, and see if they take my invitation...
Isn't that ho it works? I mean going with your analogy... Sure, you can marry someone and never touch each other or move in together. But that's kind of a weird one, in my opinion. Of course you should be able to do that. But it might require some more explicit agreement than going the default route. And I think that's what happened here. Assumptions have been made, those turned out to be wrong and now people need to find a way to deal with it so everyone's needs are met...
JustAnotherKay@lemmy.world 2 months ago
Air gapping my service is the agreement you’re talking about in this analogy, but otherwise I do actually agree with you. There is a lot of implied consent, but I think we have a near miss misunderstanding on one part.
In this scenario (analogies are nice but let’s get to reality) crawling the website to check the MAU, as harmless as it is, is still adding load to the server. A tiny amount, sure, but if you’re going to increase my workload by even 1% I wanna know beforehand. Thus, I put things on my website that say “don’t increase my workload” like robots.txt and whatnot.
Other people aren’t this concerned with their workload, in which case it might be fine to go with implied consent. However, it’s always best to follow the best practices and just make sure with the owner of a server that it’s okay to do anything to their server IMO
hendrik@palaver.p3x.de 2 months ago
I don't think that'll work. Asking for consent and retrieving the robots.txt is yet another request with a similar workload. So by that logic, we can't do anything on the internet. Since asking for consent is work and that requires consent, which requires consent... And if you're concerned with efficiency alone, cut the additional asking and complexity by just straightforward doing the single request.
Plus, it's not even that complex. Sending a few bytes of JSON with daily precalculated numbers is a fraction of what a single user interaction does. It's maybe zero point something of a request. Or with a lots of more zero's in-between if we look at what a server does each day. I mean every single refresh of the website or me opening the app loads several files, API endpoints, regularly loads hundreds of kilobytes of Javascript, images etc. There are lots of calculations and database requests involved to display several posts along with votes etc. I'd say one single pageview of me counts like the FediDB collecting sttats each day for like 1000 years.
I invented these numbers. They're wrong. But I think you get what I'm trying to say... For all practical purposes, these requests are for free and have zero cost. Plus if it's efficientcy, it's always a good idea not to ask to ask, but outright do it and deal with it while answering. So it rally can't be computational cost. It has to be consent.
JustAnotherKay@lemmy.world 2 months ago
You’re definitely right that I went a bit extreme with what I used as a reason against it, but I feel like the point still stands about “just ask before you slam people’s servers with yet another bot on the pile of millions of bots hitting their F2B system”
notfromhere@lemmy.ml 2 months ago
How is it air gapped and federated? Do you unairgap it periodically for a refresh then reairgap it? I’ve not heard of airgapped federated servers before and am intrigued. Is it purely for security purposes or also bandwidth savings? Are there other reasons one may want to run an air gapped instance?
JustAnotherKay@lemmy.world 2 months ago
In this scenario, I have multiple servers which are networked together and federated via ActivityPub but the server cluster itself is air gapped.
As to your questions about feasibility and purposes, I will admit I both didn’t think about that, and should have been more clear that this air gapped federated instance was theoretical lol
notfromhere@lemmy.ml 2 months ago
Then I’m not sure what point you were trying to make in the above conversation lol.