robots.txt is a bit like asking bots to not visit my site; with .htaccess, you’re not asking