Comment on Codeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.

<- View Parent
sudo@programming.dev ⁨2⁩ ⁨weeks⁩ ago

Places like cloudflare and akamai are already using machine learning algorithms to detect bot traffic at a network level. You need to use similar machine learning to evade them. And since most of these scrapers are for AI companies I’d expect a lot of the scrapers to be LLM generated.

source
Sort:hotnewtop