You’d probably feel differently if it were your service. Should you be able to control who scrapes your sites or should that be all or nothing?
For the record, I fucking hate what the internet is becoming. I naively believed that even if shit got cordoned off into the walled gardens that are mobile phone apps, the web would remain as open as it was. This is a terrible sign of things to come.
Eril@feddit.org 3 months ago
Actually currently it contains this:
Well, that actually is a blanket ban for everyone, so something else must be at play here.
starman@programming.dev 3 months ago
merj.com/…/investigating-reddits-robots-txt-cloak…
Reddit is serving different file to google
russjr08@bitforged.space 3 months ago
That’s real rich, coming from Reddit.