Just yesterday I banned Facebook’s user agent that is supposedly only used to generate link previews. It was being extremely aggressive, making 20 requests per second. At the time I thought “I bet they’re actually building their own search index”, and here we are.