Which is still unacceptable.
Comment on CrowdStrike downtime apparently caused by update that replaced a file with 42kb of zeroes
diffusive@lemmy.world 2 months ago
If I had to bet my money, a bad machine with corrupted memory pushed the file at a very final stage of the release.
The astonishing fact is that for a security software I would expect all files being verified against a signature (that would have prevented this issue and some kinds of attacks
LodeMike@lemmy.today 2 months ago
Angry_Autist@lemmy.world 2 months ago
From my experience it was more likely to be an accidental overwrite from human error with recent policy changes that removed vetting steps.
rozodru@lemmy.ca 2 months ago
this is what I suspect also. I mean it’s easy to point fingers at George Kurtz as he was CTO at Mcafee when they had their “little” snafu but…well…yeah. I strongly suspect many of his “policies” he had while CTO at Mcafee carried over to Crowdstrike. dude isn’t exactly known for being a fan of testing or vetting processes. in fact he’s all about quick development/crunch.
Angry_Autist@lemmy.world 2 months ago
Quick development will probably spell the end of the internet once AI code creation hits its stride. It’ll be like the most topheavy SCRUM you’ve ever seen with the devs literally incapable of disagreeing.
I was thinking about his stint at McAfee, and I think you’re right. My real question is: will the next company he golden parachutes off to learn the lesson?
I’m going to bet not.
LodeMike@lemmy.today 2 months ago
Which is still unacceptable.
BossDj@lemm.ee 2 months ago
So here’s my uneducated question: Don’t huge software companies like this usually do updates in “rollouts” to a small portion of users (companies) at a time?
Dashi@lemmy.world 2 months ago
I mean yes, but one of the issuess with “state of the art av” is they are trying to roll out updates faster than bad actors can push out code to exploit discovered vulnerabilities.
The code/config/software push may have worked on some test systems but MS is always changing things too.
madcaesar@lemmy.world 2 months ago
Somone else said this wasn’t a case of this breaks on windows system version XXX with update YYY on a Tuesday at 12:24 pm when clock is set to eastern standard time. It literally breaks on ANY windows machine, instantly, on boot. There is no excuse for this.
umbrella@lemmy.ml 2 months ago
the smart ones probably do
echodot@feddit.uk 2 months ago
Companies don’t like to be beta testers. Apparently the solution is to just not test anything and call it production ready.
JasonDJ@lemmy.zip 2 months ago
Every company has a full-scale test environment. Some companies are just lucky enough to have a separate prod environment.
Norgoroth@lemmy.world 2 months ago
Peak programmer humor
expr@programming.dev 2 months ago
That’s certainly what we do in my workplace. Shocked that they don’t.
deegeese@sopuli.xyz 2 months ago
When I worked at a different enterprise IT company, we published updates like this to our customers and strongly recommended they all have a dedicated pool of canary machines to test the update in their own environment first.
I wonder if CRWD advised their customers to do the same, or soft-pedaled the practice because it’s an admission there could be bugs in the updates.
I know the suggestion of keeping a stage environment was off putting to smaller customers.