Comment on A single point of failure triggered the Amazon outage affecting millions

DudeImMacGyver@kbin.earth ⁨3⁩ ⁨weeks⁩ ago

• rumba@lemmy.zip ⁨3⁩ ⁨weeks⁩ ago

  I certainly don’t miss dealing with air conditioning, dry fire protection, and redundant internet connections.

  I also don’t miss trying to deal with aging servers out and bringing new hardware in.

  source

  • dubyakay@lemmy.ca ⁨3⁩ ⁨weeks⁩ ago

    That work is still being done by someone in a data centre. But all these jobs went from in-house positions to the centres.

    source

    • partial_accumen@lemmy.world ⁨3⁩ ⁨weeks⁩ ago

      That work is still being done by someone in a data centre. But all these jobs went from in-house positions to the centres.

      The difference is scale. When in-house, the person responsible for managing the glycol loop is also responsible for the other CRACs, possibly the power rails, and likely the fire suppression. In a giant provider, each one of those is its own team with dozens or hundreds of people that specialize in only their area. They can spend 100% on their one area of responsibilty instead of having to wear multiple hats. The small the company, the more hats people have to wear, and the worse to overall result is because of being spread to thin.

      source
• partial_accumen@lemmy.world ⁨3⁩ ⁨weeks⁩ ago

  We need to ditch cloud entirety and go in house again.

  For many many companies that would be returning to the bad-old-days.

  I don’t miss getting an emergency page during the Thanksgiving meal because there’s excessive temperature being reported in the in-house datacenter. Going into the office and finding the CRAC failed and its now 105 degree F. And you knew the CRAC preventive maintenance was overdue and management wouldn’t approve the cost to get it serviced even though you’ve been asking for it for more than 6 months. You also know with this high temp event, you’re going to have an increased rate of hard drive failures over the next year.

  No thank you.

  source

  • corsicanguppy@lemmy.ca ⁨3⁩ ⁨weeks⁩ ago

    There’s a huge gulf between pub clowd and shitty on-prem. My daytime contract is with an organization almost completely on-prem for privacy, although on-prem to them means priv-cloud. Space has been rented. Redundant everything piped in. Redundant everything set up. We run VMs by terraform. Wheeeeee

    Point is, posing shitty on-prem as the alternative to the clowd is moving the goalposts a bit.

    source

    • partial_accumen@lemmy.world ⁨3⁩ ⁨weeks⁩ ago

      There’s a huge gulf between pub clowd and shitty on-prem.

      We agree on this.

      Redundant everything piped in. Redundant everything set up. We run VMs by terraform. Wheeeeee

      For that customer of yours, is that a single datacenter or does is represent multiple datacenters separated by a large distance across a nation, or perhaps even across national borders?

      Point is, posing shitty on-prem as the alternative to the clowd is moving the goalposts a bit.

      I think ignoring that shitty on-prem represented a large part of IT infrastructure prior cloud providers is ignoring a critical point. Was it possible to have well-run enterprise IT data centers prior to cloud? Sure. Was everyone doing that? Absolutely not, I’d argue the majority had at least a certain level of jank in their infra and that that floor is raised with cloud providers. Just the basic facilities is enterprise grade irrespective of the server or app config.

      source
• UnderpantsWeevil@lemmy.world ⁨3⁩ ⁨weeks⁩ ago

  The inverse of the old axiom “The cloud is just someone else’s computer” is “Yes, duh, that’s how you get economies of scale”.

  In-housing would mean an enormous increase in demand for physical hardware and IT technical services with a large variance in quality and accessibility. Like, it doesn’t fix the problem. It just takes one big problem and shatters it into a thousand little problems.

  source

  • dubyakay@lemmy.ca ⁨3⁩ ⁨weeks⁩ ago

    That’s good though. It means half the internet wouldn’t fail.

    source

    • UnderpantsWeevil@lemmy.world ⁨3⁩ ⁨weeks⁩ ago

      I think some of you younger folks really don’t know what the Internet was like 20 years ago.Shit was up and down all the time.

      I worked on a project back in 2008 where I had to physically haul hardware from Houston to Dallas just to keep a second rate version of a website running until we got power back at the original office. Latency at the new location was so bad that we were scrambling to reinvent the website in real time to try and improve performance. We ended up losing the client. They ended up going bankrupt. An absolute nightmare.

      Getting screamed at by clients. Working 14 hour days in a cramped server room on something way outside my scope.

      Would have absolutely killed for something as clean and reliable as AWS. Not like it didn’t even exist back then. But we self-hosted because it was cheaper.

      source
• nova_ad_vitum@lemmy.ca ⁨3⁩ ⁨weeks⁩ ago

  We don’t have to. It is entirely possible to engineer applications and services in a way that they’re not dependent on any one cloud service, while also using cloud services for IaaS. Netflix famously does this, and sure enough Netflix experience no service interruptions during this latest outage despite having a large AWS presence.

  source

  • DudeImMacGyver@kbin.earth ⁨3⁩ ⁨weeks⁩ ago

    If we want a truly robust system, yeah, we kinda do. This sort of event is only one of the issues with allowing a single entity to control pretty much everything.

    There are plenty of potential issues from a corrupt rogue corporation hijacking everything to attacks to internal fuck-ups like we just experienced. Sure, they can design a better cloud, but at the end of the day, it's still their cloud. The Internet needs to be less centralized, not more (and I don't just mean that purely in terms of infrastructure, though that is included of course).

    source

    • nova_ad_vitum@lemmy.ca ⁨3⁩ ⁨weeks⁩ ago

      If we want a truly robust system, yeah, we kinda do. This sort of event is only one of the issues with allowing a single entity to control pretty much everything.

      What I’m advocating for is the opposite of “allowing one entity to control everything”.

      en.wikipedia.org/wiki/Chaos_engineering#Chaos_Mon…

      Read about it dude. Netflix has a large presence in all major cloud providers (and they have their own data centers), but has a service whose uptime is NOT dependent on any one of those hosting environments. The proof is the pudding - Netflix service did not go down in the recent AWS outage, nor in the last one.

      source

      • DudeImMacGyver@kbin.earth ⁨2⁩ ⁨weeks⁩ ago

        Yes, Netflix had their own infrastucture in addition to other multiple redundant cloud services for their CDNs: You're kind of proving my point?

        source

        • -> View More Comments
• blackn1ght@feddit.uk ⁨3⁩ ⁨weeks⁩ ago

  Bit of an over-reaction to one incident. I’d be willing to bet the uptime, reliability and scalability of AWS is significantly better than what the vast majority of in-house solutions could do. It’s absolutely not worth going back.

  Millions of customers using AWS also weren’t affected - the company I work for certainly wasn’t, although some of our tools like Jira were.

  source

  • DudeImMacGyver@kbin.earth ⁨3⁩ ⁨weeks⁩ ago

    The problem is far more pervasive than any single incident, allowing a single megacorporation to control most of the Internet is a bad idea for many reasons.

    source

    • blackn1ght@feddit.uk ⁨3⁩ ⁨weeks⁩ ago

      Agreed, but other cloud providers exist and it would be good if there was stronger competition in this space. But going back to self hosting is a huge step back and I think if a CTO said they were going to move from the cloud back to a self hosted solution, pretty much everyone would hate it.

      source

      • DudeImMacGyver@kbin.earth ⁨3⁩ ⁨weeks⁩ ago

        There are still self hosted places today, not everything is cloud based.

        Also, there isn't more competition largely because of Amazon so, while I agree with the sentiment that it could improve things, in practice it's a moot point.

        source