I feel like most casual users would not make the connection of “crawlers” to link previews that they talk about it the article. Sure, if you understand that robots.txt includes all robots then sure. But that is not how general news media has been talking about robots.txt.