and having a bot thrashing a server indexing everything

This is a completely separate argument and one that we already have mechanisms for. Servers can use status codes and headers to warn about rate limits and block offenders.

It is also one thing to read/interact with a site as that adds value to the site as a whole

A search index adds value as well; that's why this keeps coming up. And, again, there are existing mechanisms to handle this. A robots.txt file can indicate you don't want to be crawled and offenders can be IP blocked