Comment on All of Japan's Toyota Assembly Plants Shut Down for a Day Because Their Server Ran Out of Disk Space
Swiggles@lemmy.blahaj.zone 9 months ago
This happens. Recently we had a problem in production where our database grew by a factor of 10 in just a few minutes due to a replication glitch. Of course it took down the whole application as we ran out of space.
Some things just happen and all head room and monitoring cannot save you if things go seriously wrong. You cannot prepare for everything in life and IT I guess. It is part of the job.
RidcullyTheBrown@lemmy.world 9 months ago
Bad things can happen but that’s why you build disaster recovery into the infrastructure. Especially with a compqny as big as Toyota, you can’t have a single point of failure like this. They produce over 13,000 cars per day. This failure cost them close to 300,000,000 dollars just in cars.
frododouchebaggins@lemmy.world 9 months ago
The IT people that want to implement that disaster recovery plan do not make the purchasing decisions. It takes an event like this to get the retards in the C-suite listen to IT staff.
GloveNinja@lemmy.world 9 months ago
In my experience, the C-Suite dicks will put the hammer down on someone and maybe fire a couple of folks. They’ll demand a summary of what happened and what will be done to stop it from happening again. IT will provide legit options to resolve this long term, but because that comes with a price tag they’ll be told to fix it with “process changes” and the cycle continues.
If they give IT money that’s less for themselves at EOY for bonuses so it’s a big concern /s
Swiggles@lemmy.blahaj.zone 9 months ago
Yea, fair point regarding the single point of failure. I guess it was one of those scenarios that should just never happen.
I am sure it won’t happen again though.
As I said it can just happen even though you have redundant systems and everything. Sometimes you don’t think about that one unlikely scenario and boom.