Comment

Comment on Cloudflare announces AI Labyrinth, which uses AI-generated content to confuse and waste the resources of AI Crawlers and bots that ignore “no crawl” directives.

<- View Parent

Buelldozer@lemmy.today ⁨1⁩ ⁨week⁩ ago

and try to slam your site with like 200+ requests per second

Your solution would do nothing to stop the crawlers that are operating 10ish rps. There’s ones out there operating at a mere 2rps but when multiple companies are doing it at the same time 24x7x365 it adds up.

Some incredibly talented people have been battling this since last year and your solution has been tried multiple times. It’s not effective in all instances and can require a LOT of manual intervention and SysAdmin time.

thelibre.news/foss-infrastructure-is-under-attack…

source

Sort:hotnew top

confusedbytheBasics@lemmy.world ⁨1⁩ ⁨week⁩ ago
Yep. After you ban all the easy to spot ones you’re still left with far too many hard to ID bots. At least if your site is popular and large.

source
dual_sport_dork@lemmy.world ⁨1⁩ ⁨week⁩ ago
It’s worked alright for me. Your mileage may vary.

If someone is scraping my site at a low crawl rate I honestly don’t care so long as it doesn’t impact my performance for everyone else. If I hosted anything that was not just public knowledge or copy regurgitated verbatim from the bumf provided by the vendors of the brands I sell, I might oppose to it ideologically. But I don’t. So I don’t.

If parallel crawling from multiple organizations legitimately becomes a concern for us I will have to get more creative. But thus far it hasn’t, and honestly just wholesale blocking Amazon from our shit instantly solved 90% of the problem.

source