Comment on X updates its Terms to prohibit crawling/scraping of its data
dingleberry@discuss.tchncs.de 1 year ago
Just update robot.txt coward!
Comment on X updates its Terms to prohibit crawling/scraping of its data
dingleberry@discuss.tchncs.de 1 year ago
Just update robot.txt coward!
Iwasondigg@lemmy.one 1 year ago
Took a look at their robots.txt, it appears to block all bots except Google.