Comment on FediDB has stoped crawling until they get robots.txt support
ada@lemmy.blahaj.zone 2 weeks agoGotosocial has a setting in development that is designed to baffle bots that don’t respect robots.txt. FediDB didn’t know about that feature and thought gotosocial was trying to inflate their stats.
In the arguments that went back and forth between the devs of the apps involved, it turns out that FediDB was ignoring robots.txt. ie, it was badly behaved
mesamunefire@lemmy.world 2 weeks ago
Interesting! Is this over a Git issue somewhere? That could explain quite a bit.
ada@lemmy.blahaj.zone 2 weeks ago
Yep!
Pika@sh.itjust.works 2 weeks ago
issue link here
It was a good read
mesamunefire@lemmy.world 2 weeks ago
Thank you for providing the link.