Comment on FediDB has stoped crawling until they get robots.txt support
ada@lemmy.blahaj.zone 1 year agoGotosocial has a setting in development that is designed to baffle bots that don’t respect robots.txt. FediDB didn’t know about that feature and thought gotosocial was trying to inflate their stats.
In the arguments that went back and forth between the devs of the apps involved, it turns out that FediDB was ignoring robots.txt. ie, it was badly behaved
mesamunefire@lemmy.world 1 year ago
Interesting! Is this over a Git issue somewhere? That could explain quite a bit.
ada@lemmy.blahaj.zone 1 year ago
Yep!
Pika@sh.itjust.works 1 year ago
issue link here
It was a good read
mesamunefire@lemmy.world 1 year ago
Thank you for providing the link.