Comment on I was wrong about robots.txt

<- View Parent
cecilkorik@lemmy.ca ⁨1⁩ ⁨week⁩ ago

Absolutely true. They’ll buy the data they want from some shitty crawler running from some data broker in some far-flung and lawless part of the world, hallucinate the actual source, and pretend they had no idea their “data partner” wasn’t respecting robots.txt if they have to, which they won’t ever have to do because it’s literally impossible to detect and prove and realistically unenforceable.

This is a company that removed it’s company motto of “Don’t be evil” because it found it too “limiting”. Don’t be naive.

source
Sort:hotnewtop