that is not how general news media has been talking about robots.txt.

Ahh, yes. I think there is a lesson there.