Comment

Comment on I use Zip Bombs to Protect my Server

Anyone who writes a spider that’s going to inspect all the content out there is already going to have to have dealt with this, along with about a bazillion other kinds of oddball or bad data.

source

Sort:hotnew top

catloaf@lemm.ee ⁨1⁩ ⁨day⁩ ago
Competent ones, yes. Most developers aren’t competent, scraper writers even less so.

source
- idriss@lemm.ee ⁨1⁩ ⁨day⁩ ago
  That’s true. Scrapping is a gold mine for the people that don’t know. I worked for a place which crawls the internet and beyond (fetches some internal dumps we pay for). There is no chance a zip bomb would crash the workers as there are strict timeouts and smell tests (even if a does it will crash an ECS task at worst and we will be alerted to fix that within a short time). We were as honest as it gets though, following GDPR, honoring the robots file, no spiders or scanners allowed, only home page to extract some insights.
  
  I am aware of some big name EU non-software companies very interested in keeping an eye on some key things that are only possible with scraping.
  
  source
YMS@discuss.tchncs.de ⁨11⁩ ⁨hours⁩ ago
If you have billions of targets to scan, there’s generally no need to handle each and every edge case. Just ignoring what you can’t understand easily and jumping on to the next target is an absolutely viable strategy. You will never be able to process everything anyway.

Of course, it changes a bit if some of these targets actually make your bot crash. If it happens to often, you will want to harden your bot against it. Then again, if it just happens every now and then, it’s still much easier to just restart and continue with the next target.

source
lennivelkant@discuss.tchncs.de ⁨1⁩ ⁨day⁩ ago
That’s the usual case with arms races: Unless you are yourself a major power, odds are you’ll never be able to fully stand up to one (at least not on your own, but let’s not stretch the metaphor too far). Often, the best you can do is to deterr other, minor powers and hope major ones never have a serious intent to bring you down.

In this specific case, the number of potential minor “attackers” and the hurdle for “attack” mKe it attractive to try to overwhelm the amateurs at least. You’ll never get the pros, you just hope they don’t bother you too much.

source