Comment on FediDB has stoped crawling until they get robots.txt support

<- View Parent
swizzlestick@lemmy.zip ⁨2⁩ ⁨months⁩ ago

It would be better if the server could detect bots and send them down a rabbit hole

Already possible: Nepenthes.

source
Sort:hotnewtop