Comment on FediDB has stoped crawling until they get robots.txt support
notfromhere@lemmy.ml 1 month agoThen I’m not sure what point you were trying to make in the above conversation lol.
Comment on FediDB has stoped crawling until they get robots.txt support
notfromhere@lemmy.ml 1 month agoThen I’m not sure what point you were trying to make in the above conversation lol.
JustAnotherKay@lemmy.world 1 month ago
The point was “don’t add another bot into the pile of millions of bots that hit people’s servers every day unless you’re gonna be polite about it”