I can get a 50Gb/s residential link where I am, and have a whole rack of servers.
Sounds like a good opportunity to crowd fund thousands and thousands of common scrap able instances that have random poisoning.
Comment on A Project to Poison LLM Crawlers
vane@lemmy.world 5 days ago
I have around 10-20GB github / gitlab mirror. I am constantly under attack from crawlers from top US technology corporations and LLM startups. Whenever I ban one IP range they switch to other - I don’t know if those fuckers have tickets in their systems to do it manually or they just deploy this shit all over the planet. From what I observe during attacks that I mitigate the best way to poison them is to just create gitea instance with poisoned code repository and couple hundred revisions. It’s because what they are most interested in is html representation of diff between two git revisions.
douglasg14b@lemmy.world 5 days ago
vane@lemmy.world 4 days ago
To be honest bandwidth isn’t a problem because it’s text files. The problem is to optimize network stack for multiple connections because they’re hitting from whole subnets without any delay so literally ddos and cache those html files because at some point CPU becomes bottleneck.
douglasg14b@lemmy.world 4 days ago
This is assuming aggressively cached, yes.
Also “Just text files” is what every website is sans media. And you can still, EASILY get 10+ MB pages this way between HTML, CSS, JS, and JSON. Which are all text files.
A gitea repo page for example is 400-500KB transferred (1.5-2.5MB decompressed) of almost all text.
If you have a repo with 150 files, and the scraper isn’t caching assets (many don’t) then you just served up 60MB of HTMl/CSS/JS alongside the actual repository assets.
vane@lemmy.world 4 days ago
I don’t know from theory or counting but I know that my 8 cores depleted sooner than my bandwidth and I have like 60 Mb/s uplink. My linux network stack parameters are pretty aggressive. The way I figured out that something is not right was when I heard loud fan noise from my server inside room. I logged in and all cores were red and logs were showing corporate fuckers trying to burn my house.
E_coli42@lemmy.world 5 days ago
Why isn’t there anything in the DMCA for stopping crawlers? They have stuff about requiring crawlers to follow attribution and whatnot, but nothing for not allowing crawlers in the first place. Stupid as shit.