Comment on I was wrong about robots.txt
General_Effort@lemmy.world 1 week ago
What did he think a crawler is? Why was he surprised that not allowing companies to use his data lead to them not using his data? Looks like he has another surprise coming when he notices that search engines no longer index his blog.
Archr@lemmy.world 1 week ago
I feel like most casual users would not make the connection of “crawlers” to link previews that they talk about it the article. Sure, if you understand that robots.txt includes all robots then sure. But that is not how general news media has been talking about robots.txt.
General_Effort@lemmy.world 1 week ago
Ahh, yes. I think there is a lesson there.