Based on this graph, and this graph alone, guess at what time I completely blocked OpenAI crawlers

Comments

• early_riser@lemmy.world ⁨1⁩ ⁨day⁩ ago

  It’s already hard enough for self-hosters and small online communities to deal with spam from fleshbags, now we’re being swarmed by clankers. I have a little Mediawiki to document my deranged maladaptive daydreams worldbuilding and conlanging projects, and the only traffic besides me is likely AI crawlers.

  I hate this so much. It’s not enough that huge centralized platforms have the network effect on their side, they have to drown our quiet little corners of the web under a whelming flood of soulless automata.

  source

  • wonderingwanderer@sopuli.xyz ⁨1⁩ ⁨day⁩ ago

    Anubis is supposed to filter out and block all those bots from accessing your webpage.

    Iocaine, nepenthes, and/or madore’s book of infinity are intended to redirect them into a maze of randomly generated bullshit, which still consumes resources but is intended to poison the bots’ training data.

    So pick your poison

    source

    • MonkeMischief@lemmy.today ⁨16⁩ ⁨hours⁩ ago

      Iocaine, nepenthes, and/or madore’s book of infinity are intended to redirect them into a maze of randomly generated bullshit

      We’ve officially reached a place where cyberspace is beginning to look like communing with the arcane. Lol

      source

      • -> View More Comments
  • NewNewAugustEast@lemmy.zip ⁨21⁩ ⁨hours⁩ ago

    I was up 10 to 20 percent month over month, and suddenly up 1000% it has spiked hard and they all are data harvestors.

    source
• Thorry@feddit.org ⁨1⁩ ⁨day⁩ ago

  Yeah I had the same thing. All of a sudden the load on my server was super high and I thought there was a huge issue. So I looked at the logs and saw an AI crawler absolutely slamming my server. I blocked it, so it only got 403 responses but it kept on slamming. So I blocked the IPs it was coming from in iptables, that helped a lot. My little server got about 10000 times the normal traffic.

  I sorta get they want to index stuff, but why absolutely slam my server to death? Fucking assholes.

  source

  • Ephera@lemmy.ml ⁨23⁩ ⁨hours⁩ ago

    My best guess is that they don’t just index things, but rather download straight from the internet when they need fresh training data. They can’t really cache the whole internet after all…

    source

    • Techlos@lemmy.dbzer0.com ⁨23⁩ ⁨hours⁩ ago

      Bingo, modern datasets are a list of URL’s with metadata rather than the files themselves. Every new team/individual wanting to work with the dataset becomes another DDoS participant.

      source
    • spicehoarder@lemmy.zip ⁨21⁩ ⁨hours⁩ ago

      The sad thing is that they could cache the whole internet if there was a checksum protocol.

      Now that I’m thinking about it, I actually hate the idea that there are several companies out there with graph databases of the entire internet.

      source
• punrca@piefed.world ⁨18⁩ ⁨hours⁩ ago

  It’s best to use either Cloudflare (best IMO) or Anubis.

  1. If you don’t want any AI bots, then you can setup Anubis (open source; requires JavaScript to be enabled by the end user): https://github.com/TecharoHQ/anubis

  2. Cloudflare automatically setups robots.txt file to block “AI crawlers” (but you can setup to allow “AI search” for better SEO). Eg: https://blog.cloudflare.com/control-content-use-for-ai-training/#putting-up-a-guardrail-with-cloudflares-managed-robots-txt

  Cloudflare also has an option of “AI labyrinth” to serve maze of fake data to AI bots who don’t respect robots.txt file.

  source

  • shane@feddit.nl ⁨10⁩ ⁨hours⁩ ago

    If you’re relying on Cloudflare are you even self-hosting?

    source

    • CyberSeeker@discuss.tchncs.de ⁨7⁩ ⁨hours⁩ ago

      If you build a house, buy hire a guard for the front gate, do you even own the house?!

      source
  • AHemlocksLie@lemmy.zip ⁨13⁩ ⁨hours⁩ ago

    Pretty sure I’ve repeatedly heard about the crawlers completely ignoring robots.txt, so does Cloudflare really do that much?

    source

    • Sv443@sh.itjust.works ⁨10⁩ ⁨hours⁩ ago

      Like a lock on a door, it stops the vast majority but can’t do shit about the actual professional bad guys

      source
• phase@lemmy.8th.world ⁨13⁩ ⁨hours⁩ ago

  I need to find a way to add this proof of work to my Traefik.

  source

  • hoppolito@mander.xyz ⁨12⁩ ⁨hours⁩ ago

    I ended up adding go-away in front of my code forge and anything showing dynamic info, and it turned out to be way less of a hassle than I feared with two redirects and a couple custom rules.

    If you already have traefik redirecting to your services, shouldn’t be too tough to get the extra layer of indirection added (even more so if it’s containerized).

    source
• e8CArkcAuLE@piefed.social ⁨1⁩ ⁨day⁩ ago

  that’s the kind of shit we pollute our air and water for…and properly seal and drive home the fuckedness of our future and planet.

  i totally get you se ring them to nepenthes though.

  source
• CoreLabJoe@piefed.ca ⁨1⁩ ⁨day⁩ ago

  Blocking them locally is one way, but if you’re already using cloudflare there’s a nice way to do it UPSTREAM so it’s not eating any of your resources.

  You can do geofencing/blocking and bot-blocking via Cloudflare: https://corelab.tech/cloudflarept2/

  source
• eli@lemmy.world ⁨16⁩ ⁨hours⁩ ago

  I ended up just pushing everything behind my tailnet and only leave my game server ports open(which are non-standard ports).

  source
• kambusha@sh.itjust.works ⁨1⁩ ⁨day⁩ ago

  Vendetta 1800

  source
• FukOui@lemmy.zip ⁨16⁩ ⁨hours⁩ ago

  What visualisation app is this?

  source

  • tuhriel@discuss.tchncs.de ⁨12⁩ ⁨hours⁩ ago

    Munin (munin-monitoring.org) It’s not very pretty but quite easy to setup and doesn’t eat so much resources as a Prometheus/grafana setup

    source
• x00z@lemmy.world ⁨22⁩ ⁨hours⁩ ago

  50% of my traffic is scrapers now. I really want to block them but I also want my content to be indexed and used for LLMs. At the moment there isn’t really an in-between way of doing that. :(

  (This is with me knowing they fuck up the electricity nets and memory chips, I’m just hoping that gets better soon.)

  source

  • Anarki_@lemmy.blahaj.zone ⁨21⁩ ⁨hours⁩ ago

    Why do you want your stuff in the lie machines? 🤔

    source

    • lost_screwdriver@thelemmy.club ⁨21⁩ ⁨hours⁩ ago

      That they do not become lie machines. Propaganda, lies and fake news from various different sources gets spammed all across the internet. If AI picks it up, it can just spread misinformation, especially if all trustworthy or useful sources block them

      source

      • -> View More Comments
    • x00z@lemmy.world ⁨12⁩ ⁨hours⁩ ago

      I work on a project that has a lot of older, less technical and international users who could use some extra help. We’re also not always found by the people that would benefit from our project. keeperfx.net

      source
• A_A@lemmy.world ⁨22⁩ ⁨hours⁩ ago

  vendredi à 16h30 … curieusement, personne n’essaie de répondre à ta question 😋

  source

  • hylobates@jlai.lu ⁨6⁩ ⁨hours⁩ ago

    Pile poil

    source
• JustinTheGM@ttrpg.network ⁨20⁩ ⁨hours⁩ ago

  I’m gonna guess 17:25:20

  source
• Decronym@lemmy.decronym.xyz ⁨1⁩ ⁨day⁩ ago

  Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

  Fewer Letters	More Letters
  HTTP	Hypertext Transfer Protocol, the Web
  IP	Internet Protocol
  nginx	Popular HTTP server

  ---

  [Thread #90 for this comm, first seen 13th Feb 2026, 17:40] [FAQ] [Full list] [Contact] [Source code]

  source
• quick_snail@feddit.nl ⁨20⁩ ⁨hours⁩ ago

  Y’all need to learn to cache things, shiit

  source

  • poVoq@slrpnk.net ⁨16⁩ ⁨hours⁩ ago

    This is not how things work on the modern web. Did you just wake up from a 20 year coma?

    source
  • hylobates@jlai.lu ⁨6⁩ ⁨hours⁩ ago

    There is cache but for some reason this does not affect every page I serve currently

    source

    • quick_snail@feddit.nl ⁨6⁩ ⁨hours⁩ ago

      Fix the cache.

      source
• zr0@lemmy.dbzer0.com ⁨1⁩ ⁨day⁩ ago

  And what was the reason for blocking them? What is unbearable?

  source

  • Ephera@lemmy.ml ⁨22⁩ ⁨hours⁩ ago

    They cause a huge amount of load, deteriorating the service for everyone else. I’m also guessing the time ranges in the graph, where there’s no data, is when OP’s server crashed from the load and had to restart.

    That kind of shit can easily trigger alerting and will look like a DDoS attack. I would be pissed, too, if I dropped everything to see why my server is going down and it’s not even proper criminals, but rather just some silicon valley cunts.

    source

    • zr0@lemmy.dbzer0.com ⁨17⁩ ⁨hours⁩ ago

      Thanks for your time explaining. I have multiple public facing services and I never had any issues with load just because of some crawlers. That’s why I always wonder why people get so mad at them

      source
  • GreenKnight23@lemmy.world ⁨22⁩ ⁨hours⁩ ago

    it’s pretty rare for dumbasses to point themselves out these days.

    you’re doing gods work son. keep it up!

    source