Or, you say “i am the google web crawler”, which they let past the paywall so it can be added to google.

If in not wrong, Google has a set range of IP addresses for their crawerls, so not all sites will let you through just because your UA claims to be Googlebot