It’s almost as if centralizing everything is fucking stupid.
If websites are slow for you, this is why, AWS is breaking everything
Submitted 3 weeks ago by StarvingMartist@sh.itjust.works to technology@lemmy.world
https://sh.itjust.works/pictrs/image/773556ba-92c5-4915-9a2f-f4609b14a73b.png
Comments
HertzDentalBar@lemmy.blahaj.zone 3 weeks ago
innermachine@lemmy.world 3 weeks ago
But the bean counters said it was the best idea!
cmbabul@slrpnk.net 3 weeks ago
Accountants shouldn’t be allowed to make decisions beyond their personal lives. MBAs even more so
scarabic@lemmy.world 3 weeks ago
At my work there was a big push to get things out of our own isolated data centers and into AWS because we had some services that were really very vulnerable and not equipped to serve a global distributed audience well. AWS solved that problem for us and our devs liked working with it. However now our AWS bill is so high that we’re pushing everyone to reduce it. I guess that makes sense. We’re not going back to self-hosting, just optimizing and reducing. But it sure feels like whiplash from “get everything into AWS” to “we have too much in AWS.”
HertzDentalBar@lemmy.blahaj.zone 3 weeks ago
So it sounds like rather than investing in your own systems you outsourced and now it costs far more.
Shit son that’s the same as almost every industry.
Bishma@discuss.tchncs.de 3 weeks ago
So Roblox is the only of those companies that can handle proper region failover if us-east-1 shits the bed? You young ops have gotten soft! Learn to live by ChaosMonkey or die by the Gorilla.
NaibofTabr@infosec.pub 3 weeks ago
Yeah, but proper failover and recovery requires additional infrastructure, and that costs money.
Hopefully a bunch of risk management people are writing I-told-you-so emails to C-suites right now.
homesweethomeMrL@lemmy.world 3 weeks ago
Yeah but not to worry, C-suites have pretty good filtering rules in place to never read them. Saves time, really.
Boozilla@lemmy.world 3 weeks ago
Yup, and some things which can be moved cannot be done automatically, quickly or easily…even if you are prepared. AWS is a huge suite of products and services, and there’s a lot of old legacy shit running on it. I wouldn’t punch down on the ops for this one. Cybersecurity and disaster recovery are not directly profitable, so they are almost always neglected in your average shop.
Chozo@fedia.io 3 weeks ago
So Roblox is the only of those companies that can handle proper region failover if us-east-1 shits the bed?
Either that, or the reports stopped coming in since it's a school day.
SSUPII@sopuli.xyz 3 weeks ago
Roblox doesn’t really lock down regions unless they are China. The game client can connect to any server they own. It naturally falls back to the server with the least issues without prompts and intervention.
floquant@lemmy.dbzer0.com 3 weeks ago
I think they’re the only ones doing multi-cloud. If us-east-1 shits the bed, it’s a bad day for AWS in general.
darvocet@infosec.pub 3 weeks ago
Jesus Amazon is a big enough company they should be doing their own hosting not using AWS.
tomiant@piefed.social 3 weeks ago
They’re big enough to be a utility and be nationalized!
Reverendender@sh.itjust.works 3 weeks ago
And should be
ignirtoq@fedia.io 3 weeks ago
Amazon is using their own hosting. AWS stands for Amazon Web Services. Or did I miss the joke?
kami@lemmy.dbzer0.com 3 weeks ago
I for one don’t think you missed anything
Bishma@discuss.tchncs.de 3 weeks ago
Once upon a time they insisted that Amazon had independent decision making on their providers when they were needing new infrastructure and they “always decided that AWS suited them best.”
Shockingly (/s), they stopped making that claim as soon right about the same time they started admitting that their biggest users are all under Private Pricing Agreements.
gedaliyah@lemmy.world 3 weeks ago
Maybe it was a bad idea for society to put 90% of the internet on one company’s infrastructure.
Rooty@lemmy.world 3 weeks ago
Let’s take a decentralized network and centalize it! Nothing will surely go wrong.
peoplebeproblems@midwest.social 3 weeks ago
It’s kind of telling when Amazon services like Ring, Prime Video, and Alexa failed over pretty quickly, but everything else just didnt.
There’s no conspiracy here. It’s just highlighting that Amazon could prepare for AWS outages in a region, but since everything auth related was on us-east-1, everything else went down.
Shortcuts are costly.
magnolia_mayhem@lemmy.world 3 weeks ago
Hey! Great idea! Let’s make all of the internet rely on one service from one evil company! What could go wrong?
grahamja@reddthat.com 3 weeks ago
Typical Amazon undersells everyone else to remove competition. Then after everyone has gone under reminds you it is a terrible idea. Hopefully we see businesses get burnt more and determine it is better to self host, or even better yet not need half their services on “the cloud” anyways.
DarkAri@lemmy.blahaj.zone 3 weeks ago
I think evil is too strong a word for Amazon, I think benzos is one of the nicer billionaires but I have heard the horror stories from Amazon warehouses.
ArsonButCute@lemmy.dbzer0.com 3 weeks ago
Amazon essentially killed local boilstones and then used the proceeds to become a massive megaconglomerate only challangeable by alphabet, apple, and meta in their sheer capability to control media flow.
They have spies in the form of Alexa enabled IOT devices basically everywhere. They also have the Ring camera. Recently they have stayed they will share data from Ring cameras with Palantir.
Amazon is insidious. Its evil is quiet in its spread.
The sheer size of Amazon, along with their business model, means they must employ vast swaths of people. In the US alone, over 1M people are employed by Amazon. Amazon plans to lay off half of them in the next 8 years while they shift to a robotic workforce.
These “too big to fail” companies are not evil out of some malaise for humanity. They are evil out of a cavalier disinterest in anything other than increasing the size of their hoards.
dependencyinjection@discuss.tchncs.de 3 weeks ago
Nice billionaire is an oxymoron.
PhobosAnomaly@feddit.uk 3 weeks ago
Figma balls wheeeyyyy
Kolanaki@pawb.social 3 weeks ago
Figma
of your imagination?
arschflugkoerper@feddit.org 3 weeks ago
It seems that there is not a single original thought in that head of mine.
PhobosAnomaly@feddit.uk 3 weeks ago
doesn’t matter bro, they’re your thoughts and that’s what makes them awesome 😊
Lost_My_Mind@lemmy.world 3 weeks ago
Alright…
crushes your balls with ginger root which makes them burn
SatansMaggotyCumFart@piefed.world 3 weeks ago
Why does Amazon use AWS?
I thought they had an in-house solution.
criss_cross@lemmy.world 3 weeks ago
A lot of AWS are wrappers over said in house solutions
And there’s a push to move off of those to proper AWS
thatonecoder@lemmy.ca 3 weeks ago
…and I did not even notice it, aside from the news here.
Rentlar@lemmy.ca 3 weeks ago
It really shows what a great life that we on the Fediverse are living, by making our tech independent to a degree from the largest tech providers.
Matriks404@lemmy.world 3 weeks ago
I am grateful that my past me (from few months ago) decided to study Slavic philology, instead of getting stupid tech job. I hate modern tech, anything from big corporations.
I am also glad that I’ve switched to Linux (Debian ftw) to escape software enshittification.
magnolia_mayhem@lemmy.world 3 weeks ago
It’s not a full escape, just a path to escape.
luciferofastora@feddit.org 3 weeks ago
I’d argue you cna never fully escape the problem of enshittification, unless you produce and maintain everything you use by yourself. Given how much of what we need today requires specialist labour and how impossible it is to be a specialist in everything, chances are you’d be trading enshittification for plain shit.
But using Debian at least shuts down one source of enshittification. Just because perfection can’t ever fully be attained is no reason not to try and get as close as you can.
thepompe@ttrpg.network 3 weeks ago
The cloud was a scam for people with more money than sense.
Most people hosting things on the cloud, like Fediverse services, would be better off selfhosting and buying the hardware themselves.
artyom@piefed.social 3 weeks ago
Self hosting requires an immense amount of specialized knowledge and time…
thepompe@ttrpg.network 3 weeks ago
It really doesn’t, and certainly not more than running something on AWS.
In fact, hosting on cloud infrastructure adds layer of complexity.
I stand by my original assertion.
DaTingGoBrrr@lemmy.world 3 weeks ago
I got started self-hosting last week when I got ahold of a smal Lenovo ThinkCentre. I installed Proxmox on it and if I want to self-host something I just spin up a container or virtual machine on the Proxmox system. It’s so much easier than installing self-hosted projects on bare metal. And if you want to change things around then just disable or delete the container/vm and let Proxmox stay clean. If one container breaks the rest of the system will still function.
I could easily host Lemmy from home with Proxmox and a reverse proxy with my current setup. I am not going to because I am not interested in moderating a platform and all the responsibility that comes with it, but it’s very possible to do.
SCmSTR@lemmy.blahaj.zone 3 weeks ago
What are these random fucking websites though
StarvingMartist@sh.itjust.works 3 weeks ago
I mean those are giant websites and organizations that exist in the mainstream, instructure is used by most schools nowadays
dependencyinjection@discuss.tchncs.de 3 weeks ago
Which is random? I’ve only not heard of three: Instructure, fanduel, and chime.
All the ones I’ve heard of are quite massive companies and I would assume the others are too.
Rai@lemmy.dbzer0.com 3 weeks ago
Bottom left
figma ballz hahaha gotem
Bluefalcon@discuss.tchncs.de 3 weeks ago
Shit, none of my websites are slow. Fuck Amazon
cheers_queers@lemmy.zip 3 weeks ago
even the websites for local health systems have been going down. At this point if we have an event like Y2K was supposed to be, we are so fucked lol
sommerset@thelemmy.club 3 weeks ago
Blah blah blah blah blah. And nothing is going to be done about it. Keep yapping. In a week no one is going to remember. By eow no one is going to remember.
StarvingMartist@sh.itjust.works 3 weeks ago
Are you okay?
magnolia_mayhem@lemmy.world 3 weeks ago
No.
minorkeys@lemmy.world 3 weeks ago
Diversity is a strength.
dan69@lemmy.world 3 weeks ago
I’m okay to blame bozos.
SlartyBartFast@sh.itjust.works 3 weeks ago
Are any lemmy instances run on aws? Been getting a lot less articles last couple days
Typotyper@sh.itjust.works 3 weeks ago
Maybe the bots are run by AWS
JackbyDev@programming.dev 3 weeks ago
I’d be surprised if none are run in AWS.
atmorous@lemmy.world 3 weeks ago
Time for open source Amazon app alternatives to be made. Made a post on OpenSource with suggestions on how to do it and how it can help open source communities: KDE, GNU/Linux, etc can each make their own and use their profits for their own and other open source projects. A passive income if you will from making, and managing the app that displays products from various partnered stores and websites
Buddahriffic@lemmy.world 3 weeks ago
For some context, the issue was affecting AWS or Amazon Web Services, which is hosting for those other services (or parts of them), not an amazon app issue cascading to other services.
And I don’t think an open source amazon app would work. Amazon is a lot more than just a webstore. They’ve got a massive logistics network plus warehouses and packing plants. I feel like an open source version of amazon’s store would end up avoiding the corporate shit but would have all the negatives of amazon store without many of the positives.
Ferrous@lemmy.ml 3 weeks ago
Heh. Say what you will about Roblox (plenty of stuff), but seems like their infrastructure is on lock.
tomiant@piefed.social 3 weeks ago
Not my ancestors!
mesamunefire@piefed.social 3 weeks ago
Amazon sneases, the whole internet catches a cold.
Im still seeing services up/down the entire day at work. Services that are not even AWS like Azure are slow for some reason (probably businesses failing over to other infa). Its crazy.
None of our in office infa is having issues. Managers are talking about fail-overs all day lol.
tomiant@piefed.social 3 weeks ago
When a handful of people own all the companies in the world, the whole world becomes a single point of failure.
PoopMonster@lemmy.world 3 weeks ago
Soon: "Welcome to Amazon, I love you.
Boozilla@lemmy.world 3 weeks ago
There are some downstream / knock on effects going on which can be explained…but I can’t help but wonder if today’s story is bigger than just AWS. AWS saying it was an outage of a “few hours” for DynamoDB and DNS…and that doesn’t line up all that great with what people are reporting in the wild . I’m not trying to start a conspiracy theory, just wondering what the post mortems will tell us, if anything. Obviously the suits want to keep embarrassing fuckups downplayed as much as possible.
mesamunefire@piefed.social 3 weeks ago
Its crazy, we are seeing unrelated services stop sending emails, issues with DNS, all sorts of strange stuff.
Rooster326@programming.dev 3 weeks ago
They want to keep the news of the rally over the weekend as quiet as possible.
Carrot@lemmy.today 3 weeks ago
As someone with a little insider baseball knowledge, it was just a few hours down of DynamoDB and DNS. However, that caused EC2 to go down for ~1 day, which causes pretty much 1/3 of the internet to go down. Once EC2 sorts themselves out, then teams/companies (almost all amazon services use EC2 in the back end) that use EC2 have to get their ducks back in a row, and that can take any span of time, depending on how well their code was written to handle failures + how many people they are willing to pay oncall/overtime.
Boozilla@lemmy.world 3 weeks ago
Dave explains the “long tail” of recovery:
youtu.be/KFvhpt8FN18
Tollana1234567@lemmy.today 3 weeks ago
games were affected online, plus apps, and then anyone in retail who does inventory, order writing.