Same. I thought there would be more stuff happening in the background but when I saw it’s just hijacking the google bot headers to display the html i was a bit disappointed it’s so stupidly easy.
Comment on 13 Feet Ladder
redcalcium@lemmy.institute 1 year ago
It amazes my that all it takes is just changing user agent to Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) and it can bypass paywalls on many sites? I thought those sites would try harder *e.g. checking if the ip address is truly belong to google", but apparently not.
aniki@lemm.ee 1 year ago
andrew@radiation.party 1 year ago
Checking ip ownership is a moving target more likely to result in outcomes these sites don’t want (accidentally blocking google bots and preventing results from appearing on google).
Checking useragent is cheap, easier, unlikely to break (for this purpose, anyway) and the percentage of folks who know how to bypass this check is relatively slim, with a pretty small financial impact.
douglasg14b@lemmy.world 1 year ago
It’s not necessarily a moving target when entire blocks can be associated with Google.
andrew@radiation.party 1 year ago
Unless they are permanently only using specific addresses or blocks and will never change that up, I’d consider it a moving target.
efstajas@lemmy.world 1 year ago
Google literally has an official list of IP ranges for their crawlers, complete with an API that returns the current IP ranges that you can use to automate a check. Hardly a moving target, and even if it is, it doesn’t matter if you know exactly where the target is at all times.