Comment on FediDB has stoped crawling until they get robots.txt support

<- View Parent
ada@lemmy.blahaj.zone ⁨2⁩ ⁨weeks⁩ ago

Gotosocial has a setting in development that is designed to baffle bots that don’t respect robots.txt. FediDB didn’t know about that feature and thought gotosocial was trying to inflate their stats.

In the arguments that went back and forth between the devs of the apps involved, it turns out that FediDB was ignoring robots.txt. ie, it was badly behaved

source
Sort:hotnewtop