Comment on FediDB has stoped crawling until they get robots.txt support
notfromhere@lemmy.ml 10 months agoThen I’m not sure what point you were trying to make in the above conversation lol.
Comment on FediDB has stoped crawling until they get robots.txt support
notfromhere@lemmy.ml 10 months agoThen I’m not sure what point you were trying to make in the above conversation lol.
JustAnotherKay@lemmy.world 10 months ago
The point was “don’t add another bot into the pile of millions of bots that hit people’s servers every day unless you’re gonna be polite about it”