Why is serveral instances down simultaneously? Is it just me?

Comments

• kid2908@lemmynsfw.com ⁨1⁩ ⁨year⁩ ago

  Here is the reason for lemmy.fmhy.ml

  very.bignutty.xyz/notes/9hf13it1ced3b2za

  • .ml domains (the one fmhy.ml was on) has been reclaimed by the Mali government
  • Freenom is also being sued by Meta (and has been for the past few months)
  • Both of these have resulted in fmhy, along with a lot of other domains, to be unresolvable
  • Changing domains will cause us to have to refederate and start mostly from scratch (although we might be able to transfer posts and users)

  source

  • kid2908@lemmynsfw.com ⁨1⁩ ⁨year⁩ ago

    And it seem to getting worse for lemmy fmhy

    very.bignutty.xyz/notes/9hg4dquksvbha67h

    • All services, except Lemmy, is up and running again via the new domain: fmhy.net
    • Lemmy itself will require a significant database cleanup to get users transferred, posts and communities may not be possible
    • Cleanup will have to be done either manually or with a custom script

    And now to answer some common questions I’ve seen floating around:

    • Why is lemmy.ml not affected? WE DON’T KNOW. My assumption is that it’s popular (or lucky enough) to not be affected by this change. The only difference between our two domains is that lemmy.ml doesn’t use CF, and AFAIK, multiple other domains (with and without cf) are still unreachable.
    • Why can’t you just change the domain? That’s not how federation works. Most services (including Mastodon and Lemmy) do not support changing URL properly, as doing so could potentially break the whole network.
    • Why is it taking so long to get back up? Not all staff members have access to the server, CF, or domain registrar. Not to mention that this is a complicated task that requires a lot of effort, and one that we didn’t even know we had to do until the following morning. (We were under the assumption that the .ml TLD didn’t just explode.)

    source
  • kmkz_ninja@lemmy.world ⁨1⁩ ⁨year⁩ ago

    What is Freenom and why are we supposed to know what that is?

    source

    • floridaman@lemmy.blahaj.zone ⁨1⁩ ⁨year⁩ ago

      Freedom in a domains registrar that gives out ‘free’ domains for up to a year. They were the registrar for .ml domains until this happened.

      source
• badmin@lemm.ee ⁨1⁩ ⁨year⁩ ago

  sh.itjust.works admin via matrix:

  “looks like a bunch of instances are under attack at the moment”

  source

  • Shadow@lemmy.ca ⁨1⁩ ⁨year⁩ ago

    Not an attack.

    See github.com/LemmyNet/lemmy/issues/3649

    source
• OverfedRaccoon@lemmy.world ⁨1⁩ ⁨year⁩ ago

  Not saying it was a coordinated attack, per your edit, but anything popular is a prime target for various types of attacks, especially DDoS… But with every attack, the developers and various admins and owners of instances learn something new and how to mitigate it. So while it’s annoying, it’s just as much a blessing as a curse - better to patch things quick than leave an exploitable hole open for who knows how long.

  source

  • Linuturk@lemmy.onitato.com ⁨1⁩ ⁨year⁩ ago

    I run my own single user instance, and it was down as well. Not sure why someone would target a single user instance. Not ruling it out, but it seems unlikely.

    source

    • ApathyTree@lemmy.dbzer0.com ⁨1⁩ ⁨year⁩ ago

      I’m thinking about doing this for making my main account, since I never look at my local feed anyway.

      How’s the experience? And do you know of a good starting point?

      source

      • -> View More Comments
    • mrmanager@lemmy.today ⁨1⁩ ⁨year⁩ ago

      All instances using federation are publicly visible and it’s simple to script attacking all of us.

      source
• YoungPrinceAmmon@lemmy.world ⁨1⁩ ⁨year⁩ ago

  Yeah, for me too. Seems like whole fediverse has issues, problems with loading posts etc. at least for me

  source
• Blaze@iusearchlinux.fyi ⁨1⁩ ⁨year⁩ ago

  Same here, sh.itjust.works is currently down:

  www.isitdownrightnow.com/sh.itjust.works.html

  source

  • dandroid@dandroid.app ⁨1⁩ ⁨year⁩ ago

    sh.itdont.work

    source

    • Blaze@iusearchlinux.fyi ⁨1⁩ ⁨year⁩ ago

      It does most of the time, to be honest

      source

      • -> View More Comments
    • OhNoMyInstanceIsDown@lemm.ee ⁨1⁩ ⁨year⁩ ago

      sh.itjustdoesn’tfucking.work

      source
  • badmin@lemm.ee ⁨1⁩ ⁨year⁩ ago

    Which is not explained by .ml fiasco, it is worth pointing out.

    source
• hitagi@ani.social ⁨1⁩ ⁨year⁩ ago

  There is a GitHub issue on it and I experienced the exact same thing with my instance. A timeout occurs and the only way to fix it is to restart it seems. Like everyone else, it’s strange that it all happened at the same time.

  source

  • zalack@kbin.social ⁨1⁩ ⁨year⁩ ago

    It's not that strange. A timeout occurs on several servers overnight, and maybe a bunch of Lemmy instances are all run in the same timezone, so all their admins wake up around the same time and fix it.

    Well it's a timeout, so by fixing it at the same time the admins have "synchronized" when timeouts across their servers are likely to occur again since it's tangentially related to time. They're not likely to all fail again at the same moment.

    It's kind of similar to the thundering herd where a bunch of things getting errors will synchronize their retries in a giant herd and strain the server. It's why good clients will add expenses back off AND jitter (a little bit of randomness to when the retry is done, not just every x^2 seconds). That way if you have a million clients, it's less likely that all 1,000,000 of them will attempt a retry at the extract same time, because they all got an error from your server at the same time when it failed.

    source

    • hitagi@ani.social ⁨1⁩ ⁨year⁩ ago

      Interesting. Never thought of it that way.

      source
    • Blaze@iusearchlinux.fyi ⁨1⁩ ⁨year⁩ ago

      Interesting

      source
  • badmin@lemm.ee ⁨1⁩ ⁨year⁩ ago

    wrong issue lol

    source

    • hitagi@ani.social ⁨1⁩ ⁨year⁩ ago

      This probably makes more sense although the issue I was experiencing earlier had similar logs as the issue I linked and others have commented on it too around the same time. I’m guessing they’re related.

      source

      • -> View More Comments
• ryry1985@lemmy.world ⁨1⁩ ⁨year⁩ ago

  I had that too. Tried multiple instances myself.

  source
• rglullis@communick.news ⁨1⁩ ⁨year⁩ ago

  For those affected by these outages in the larger servers and who’d be interested in helping spread around: my instance will be free for the first 250 registrations. There are ~220 spots still left.

  The catch is that my registration process is (purposefully) difficult to avoid squatters/spammers/bots. So you need to do one of the following:

  • sign up to my main portal site first. If you do that, please give your email address so that I can send a confirmation with the first credentials.
  • signup to communick.news directly, and send me a DM on reddit (/u/rglullis), or here, or on mastodon) with the username you used to signup. If you don’t send the DM, I will assume it’s a bot and will deny the application.

  source

  • Blaze@iusearchlinux.fyi ⁨1⁩ ⁨year⁩ ago

    Thanks for sharing this, don’t mind the downvotes

    source
• C3D@lemmy.world ⁨1⁩ ⁨year⁩ ago

  It seem like it’s a bug which spread through more than a ddos.

  Restarting the backend of an instance should fix the problem otherwise web users could try clearing the cookies

  source
• ArchmageAzor@lemmy.world ⁨1⁩ ⁨year⁩ ago

  Another DDOS? There’s been a few of them lately

  source

  • Blaze@iusearchlinux.fyi ⁨1⁩ ⁨year⁩ ago

    Probably a sign that the platform is getting more popular, nobody wants to DDoS a place without any users

    source
• donut4ever@lemm.ee ⁨1⁩ ⁨year⁩ ago

  Only one working for me is lemme.ee. Lemmy.world and sh.itjust.works don’t work

  source

  • YoBuckStopsHere@lemmy.world ⁨1⁩ ⁨year⁩ ago

    Lemmy.world was only down for ten minutes

    source

    • donut4ever@lemm.ee ⁨1⁩ ⁨year⁩ ago

      Everything is back up now. I used connect for Lemmy and it actually tells you why an instance is not working. Lemmy.world was under maintenance, and the other one was just down.

      source
• flyingcloud11@lemmy.world ⁨1⁩ ⁨year⁩ ago

  That’s the problem with the fediverse. They tend to go down a lot. Same thing happens with mastodon servers.

  source

  • master5o1@lemmy.nz ⁨1⁩ ⁨year⁩ ago

    So does reddit.

    source
  • Blaze@iusearchlinux.fyi ⁨1⁩ ⁨year⁩ ago

    That’s fine by me. Opportunity to do something else.

    source

    • morganpurr@lemmynsfw.com ⁨1⁩ ⁨year⁩ ago

      But infinite contenttttt engagementtttt shareholderssssd

      source
• whiskers@lemmy.world ⁨1⁩ ⁨year⁩ ago

  I was facing issues with lemmy.world too, now seems to be okay

  source
• freamon@endlesstalk.org ⁨1⁩ ⁨year⁩ ago

  I get that message pretty much every time I visit a Community that I haven’t visited before (or maybe one that no-else from my instance has visited before).

  It get fixed on a refresh (like the message suggests), so I’m guessing it’s a problem with time-out settings when my instance has to pull in a lot of new data.

  source
• jsveiga@feddit.nl ⁨1⁩ ⁨year⁩ ago

  When vlemmy.net disappeared (and it was the only one I had registered to), it registered at lemmy.world, sh.itjust.works, and while thinking I was registering at lemmy.ML, I registered at lemmy.NL. (then I requested a login at lemmy.ml, but never got a confirmation).

  At the moment lemmy.NL is the only instance I have a login at that I can use.

  That was a happy mistake :-)

  Maybe the problem is me. If lemmy.nl goes down, I’ll know for sure.

  source

  • Blaze@iusearchlinux.fyi ⁨1⁩ ⁨year⁩ ago

    Ha ha ha

    source
• null@zerobytes.monster ⁨1⁩ ⁨year⁩ ago

  Overloaded servers, not enough bandwidth, endless possibillities…

  source