Comment on I was wrong about robots.txt

<- View Parent
General_Effort@lemmy.world ⁨1⁩ ⁨week⁩ ago

That’s very different from what I called false.

What you describe may happen, but probably not as much as you think. Much of that stuff is just not that valuable. Some personal, colloquial writing is necessary, but Google already pays Reddit. Other stuff is better obtained from torrents or shadow libraries like Anna’s Archive.

source
Sort:hotnewtop