Comment on I was wrong about robots.txt

<- View Parent
ell1e@leminal.space ⁨1⁩ ⁨month⁩ ago

And allowing the public crawler might also have it feed their AI: arstechnica.com/…/cloudflare-wants-google-to-chan…

source
Sort:hotnewtop