Comment on Cloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.

<- View Parent
fuckwit_mcbumcrumble@lemmy.dbzer0.com ⁨1⁩ ⁨week⁩ ago

Cloudflare isn’t the best at blocking things. As long as your crawler isn’t horribly misconfigured you shouldn’t have much issues.

source
Sort:hotnewtop