Comment on Based on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlers

zr0@lemmy.dbzer0.com ⁨1⁩ ⁨day⁩ ago

• Ephera@lemmy.ml ⁨1⁩ ⁨day⁩ ago

  They cause a huge amount of load, deteriorating the service for everyone else. I’m also guessing the time ranges in the graph, where there’s no data, is when OP’s server crashed from the load and had to restart.

  That kind of shit can easily trigger alerting and will look like a DDoS attack. I would be pissed, too, if I dropped everything to see why my server is going down and it’s not even proper criminals, but rather just some silicon valley cunts.

  source

  • zr0@lemmy.dbzer0.com ⁨18⁩ ⁨hours⁩ ago

    Thanks for your time explaining. I have multiple public facing services and I never had any issues with load just because of some crawlers. That’s why I always wonder why people get so mad at them

    source

    • hoppolito@mander.xyz ⁨33⁩ ⁨minutes⁩ ago

      I’m providing hosting for a few FOSS services, relatively small scale, for around 7 years now and always thought the same for most of that time. People were complaining about their servers being hit but my traffic was alright and the server seemed bulky enough to have a lot of buffer.

      Then, like a month or two ago, the fire nation attacked the bots came crawling. I had sudden traffic spikes of up to 1000x, memory was hogged and the CPU could barely keep up. The worst was the git forge, public repos with bots just continuously hammering away at diffs between random commits, repeatedly building out history graphs for different branches and so on - all fairly intense operations.

      After the server went to its knees multiple times over a couple days I had to block public access. Only with proof of work in front could I finally open it again without destroying service uptime. And even weeks later, they were still trying to get at different project diffs whose links they collected earlier, it was honestly crazy.

      source

      • zr0@lemmy.dbzer0.com ⁨5⁩ ⁨minutes⁩ ago

        That’s very interesting, as if only certain types of content get crawled. May I know what kind of software you used and if you had a reverse proxy in front of it?

        source
• GreenKnight23@lemmy.world ⁨1⁩ ⁨day⁩ ago

  it’s pretty rare for dumbasses to point themselves out these days.

  you’re doing gods work son. keep it up!

  source