That’s counting on one machine using the same cookie session continuously, or they code up a way to share the tokens across machines. That’s now how the bot farms work
That’s counting on one machine using the same cookie session continuously, or they code up a way to share the tokens across machines. That’s now how the bot farms work
deffard@lemmy.world 1 day ago
It will obviously depend heavily on the type of bot crawling, but that is not hard coordination for harvesting data for LLM’s, as they will already have strategies to prevent nodes all crawling the same thing - a simple valkey cache can store a solved JWT.
scrubbles@poptalk.scrubbles.tech 1 day ago
but the vast majority of crawlers don’t care to do that. That’s a very specific implementation for this one problem. I actually did work at a big scraping farm, and if they encounter something like this,they just give up. It’s not worth it to them. That’s where the “worthiness” check is, you didn’t bother to do anything to gain access.