Comment on Google Search is losing its 'cached' web page feature

<- View Parent
megaman@discuss.tchncs.de ⁨9⁩ ⁨months⁩ ago

At least some of these tools change their “user agent” to be whatever google’s crawler is.

When you browse in, say, Firefox, one of the headers that firefox sends to the website is “I am using Firefox” which might affect how the website should display to you or let the admin knkw they need firefox compatibility (or be used to fingerprint you…).

You can just lie on that, though. Some privacy tools will change it to Chrome, since that’s the most common.

Or, you say “i am the google web crawler”, which they let past the paywall so it can be added to google.

source
Sort:hotnewtop