Comment on Anubis is awesome and I want to talk aout it
sudo@programming.dev 5 hours agoIts like you didn’t understand anything I said. Anubis does work. I said it works. But it works because most AI crawlers don’t have a headless browser to solve the PoW. To operate efficiently at the high volume required, they use raw http requests. The vast majority are probably using basic python requests module.
You don’t need PoW to throttle general access to your site and that’s not the fundamental assumption of PoW. PoW assumes (incorrectly) that bots won’t pay the extra flops to scrape the website. But bots are paid to scape the website users aren’t. They’ll just scale horizontally and open more parallel connections. They have the money.
poVoq@slrpnk.net 4 hours ago
You are arguing a strawman. Anubis works because because most AI scrapers (currently) don’t want to spend extra on running headless chromium, and because it slightly incentivises AI scrapers to correctly identify themselves as such.
Most of the AI scraping is frankly just shoddy code written by careless people that don’t want to ddos the independent web, but can’t be bothered to actually fix that on their side.
sudo@programming.dev 4 hours ago
WTF, That’s what I already? That was my entire point from the start!? You don’t need PoW to force headless usage. Any JavaScript challenge will suffice. I even said the Meta Refresh challenge Anubis provides is sufficient and explicitly recommended it.
poVoq@slrpnk.net 4 hours ago
And how do you actually check for working JS in a way that can’t be easily spoofed? Hint: PoW is a good way to do that.
Meta refresh is a downgrade in usability for everyone but a tiny minority that has disabled JS.
sudo@programming.dev 4 hours ago
Accessing the browsers API in any way is way harder to spoof than some hashing. I already suggested checking if the browser has graphics acceleration. That would filter out the vast majority of headless browsers too. PoW is just math and is easy to spoof without running any JavaScript. You can even do it faster than real JavaScript users something like Rust or C.
What are you talking about? It just refreshes the page without doing any of the extra computation that PoW does. What extra burden does it put on users?