Comment on Google Is the Only Search Engine That Works on Reddit Now Thanks to AI Deal

<- View Parent
squidspinachfootball@lemm.ee ⁨3⁩ ⁨months⁩ ago

iirc, isn’t robots.txt more of a gentlemen’s agreement? I vaguely recall bots being able to crawl a site regardless, it’s just that most devs respect robots.txt and don’t. Could be wrong though, happy to be corrected.

source
Sort:hotnewtop