Submitted 3 months ago by TimeSquirrel@kbin.melroy.org to technology@lemmy.world
https://www.youtube.com/watch?v=4yDm6xNeYas
Cybersecurity firm Crowdstrike pushed an update that caused millions of Windows computers to enter recovery mode, triggering the blue screen of death. Learn ...
Are there really a billion systems in the world that run Crowdstrike? That seems implausible.
Probably includes a bunch of virtual machines.
Yeah, our VMs completely died at work. Has to set up temporary stuff on hardware we had laying around today. Was kinda fun, but stressful haha.
I doubt it’s too much of a stretch, since even here in australia, we’ve had multiple airlines, news stations, banks, supermarkets and many others, including the aluminium extrusion business my father works at, all go down, scale this do hundreds of countries with populations tenfold of ours, it puts it into perspective that there may even be more than a billion machines affected
Despite how it may seem on Lemmy, most people have not yet actually switched to Linux. This stat is legit.
I know that Windows is everywhere, I just don’t know the percentage of Windows computers that run Crowdstrike.
Yes
Sounds pretty plausible to me. An organization doesn’t have to be very big to get into the hundreds or thousands of devices on a network when you account for servers and VM.
A company with 40 employees all accessing and RDS server using a company laptop is looking at 85+ devices already
Also:
Crowd strike should be held responsible, and with that I don’t mean the developmers who were forced to do this shit, I mean the ceo, the CTO.
Jail them.
If you are so critical you better not fuck around and I can guarantee you, they were fucking around, pushing bad practices, etc. why do I say that? Because its lways like that
That comp ay should be dissolved, the C suite jailed.
Also, STOP USING WINDOWS FOR DESKTOP FOR FRACK SAKE. Switch to Linux already, I’m getting tired of having to read this shit.
If you’re using windows for servers then you deserve your place right next to those C suite guys and gals
You make Linux users look bad. Please cease.
To be fair this seems to be the sentiment on most Linux and linux-ancillary forums.
Which while wrong and ignorant on multiple levels, seems on brand none the less.
Why?
Seriously, why?
Linux is a free system with a fraction of the daily / weekly issues that Microsoft has. Its Been like this for literally decades now.
Microsoft Sella expensive paid systems that spy on us and still feed us advertising. This week, Microsoft and vendors caused likely billions of dollars in damage. Will there be any consequences? Nah. Even better; If I say stip using Microsoft software, I’m a bad guy!
Just a few months ago, it came out that Microsoft consciously decided not to fix critical security hugs resulting in hacks in the US government bu the Chinese government. There was a senate hearing in this where they weally weally promised this time they would behave. I said the same back then, install Linux already and got the same responses, I’m making Linux look bad!
So I ask you… HOW? How exactly am I the bad guy here, why isn’t everyone shitting on Microsoft and it’s providers for fucking this up so so wonderfully bad, AGAIN…?
And mind you, we all pay for this shite. Why hasn’t anyone there hone to jail for causing shit like this by making decisions that obviously knowingly would cause this shit?
But yeah, you’re riiiiight. I make Linux look bad, and we wouldn’t want that, now would we?
Sure, throw people in jail who haven’t committed a crime, that’ll fix all kinds of systemic issues
If you make decisions (typically focussing on profit over anything else) that causes so much disruption, time, and money (not to mention the possibility of risking lives), then yeah, that is a crime.
As always, if I do something like that, I get jailed. If a CTO causes it,.it’s cost of business,.let’s hand slap the company, and act as if nothing happens. Fuck them, you get paid for this. You fucked it up, you get to he actually be held responsible.
Crowdstrike took Debian and Rocky down earlier this year due to a bad update… Linux is not immune.
And?
Debian is a FREE (as in beer) AND a free (not as in beer but as in freedom) system maintained mainly by volunteers which has an actual focus on us, the end users.
Microsoft, on the other hand, makes us pay through the nose for shit systems that all have focus on Microsoft, NOT on the end user. If you make me pay and spy on me and serve me ads, then at the very fracking least I expect you to take responsibility when you fuck up, and paye for my lost time and money. However, as windows fracks up just about every week, Microsoft would be bankrupt within a month if they’d have to do that.
Keep dreaming. Maybe next year Linux on desktops will increase by another 0.2% and you can post hundreds of articles about how the “age of Linux” is coming… again.
There is no learning, companies just move to different antivirus. The new hotness, the cycle repeats over and over until the new antivirus does this same shit. Look at McAfee in 2010, in fact the CEO of Crowdstrike was the CTO of McAfee then. That easily took down millions of windows XP machines.
in fact the CEO of Crowdstrike was the CTO of McAfee then
The hero of Linux adoption then. All hail - what’s the name of that guy?
This isn’t the Windows L you think it is. This can and has happened on Linux. It’s a Crowdstrike/Bad corp IT issue.
There is learning here.
As companies, we put faith in an external entity with goals not identical to our own: a lot of faith, and a lot of control.
That company had the power to destroy our businesses, cripple travel and medicine and our courts, and delay daily work that could include some timely and critical tasks.
This is not crowdstrike’s fault; for the bad code yes, but for the indirect effects of that no. We knew - please tell me we had the brains god gave a gnat and we knew - that putting so much control in the hands of outsiders not concerned or aware of our detailed needs and priorities, was a negligent and foolish thing to do.
The lesson is to do our jobs: we need to ensure we have the ability to make the decisions to which were entrusted, and the power that authority gives us that our decisions when accepted are not threatened by a negligent mistake so boneheaded it’s all but the whim of a simpleton. We cannot choose to manage our part of our organization effectively, no matter how (un)important that organization or part is, and then share control with a force that we’ve seen can run roughshod over it.
It’s exactly like the leopards eating our face, except people didn’t see they were leopards. No one blames the leopards, as they’re just confirming to their nature, eventually.
And no one should blame this company for a small mistake, just because we let the jaws get so close to our faces that we became complacent.
Have you never worked in corporate IT or something? Of course we should blame Crowdstrike, that way we don’t get a sev 1 on our scorecard.
It’s funny that corporate IT will be one of the groups getting the blame in this case, despite it being in most cases not their decision that a company lacks a separate test and production environment. The executives that decided that usually gets off scot free.
That company had the power to destroy our businesses, cripple travel and medicine and our courts, and delay daily work that could include some timely and critical tasks.
Unless you have the ability and capacity to develop your own ISA/CPU architecture, firmware, OS, and every tool you use from the ground up, you will always be, at some point, “relying on others stuff” which can break on you at a moments notice.
That could be Intel, or Microsoft, or OpenSSH, or CrowdStrike^0. Very, very, very few organizations can exist in the modern computing world without relying on others code/hardware (with the main two that could that come to mind outside smaller embedded systems being IBM and Apple).
I do wish that consumers had held Microsoft more to account over the last few decades to properly use the Intel Protection Rings (if the CrowdStrike driver were able to run in Ring 1, then it’s possible the OS could have isolated it and prevented a BSOD, but instead it runs in Ring 0 with the kernel and has access to damage anything and everything) — but that horse appears to be long out of the gate (enough so that X86S proposes only having Ring 0 and Ring 3 for future processors).
But back to my basic thesis: saying “it’s your fault for relying on other peoples code” is unhelpful and overly reductive, as in the modern day it’s virtually impossible to do so. Even fully auditing your stacks is prohibitive. There is a good argument to be made about not living in a compute monoculture^1; and lots of good arguments against ever using Windows^2 (especially in the cloud) — but those aren’t the arguments you’re making. Saying “this is your fault for relying on other peoples stuff” is unhelpful — and I somehow doubt you designed your own ISA, CPU architecture, firmware, OS, network stack, and application code to post your comment.
——-
^0 — Indeed, all four of these organizations/projects have let us down like this; Intel with Spectre/Meltdown, Microsoft with the 28 day 32-bit Windows reboot bug, and OpenSSH just announced regreSSHion.
^1 — My organization was hit by the Falcon Sensor outage — our app tier layers running on Linux and developer machines running on macOS were unaffected, but our DBMS is still a legacy MS SQL box, so the outage hammered our stack pretty badly. We’ve fortunately been well funded to remove our dependency on MS SQL (and Windows in general), but that’s a multi-year effort that won’t pay off for some time yet.
^2 — my Windows hate is well documented elsewhere.
And for the 451855528th time: switch to Linux already. Why do people keep paying for this shit? Every time I get excuses. I switched to a Linux desktop 20 years ago. There were enout moments that I needed to tweak things to make it work but for the last decade, I haven’t had any issues.
If you’re dum enough to use windows for servers then you just deserve to burn, if you make that decision then its all on you.
This has nothing to do with Linux. Crowdstrike has in fact broken Linux installs in a fairly similar way before.
Don’t worry, if it had broken in Linux, these same posters would be railing on CrowdStrike directly, but since it broke on Windows, onvisoult Microsoft is to blame.
You make Linux users look bad
You didnt say he was wrong though.
They wouldn’t if they were consistent and had also left degenerate social media (which Lemmy is part of, despite being much better than corporate alternatives). But then they also wouldn’t because we wouldn’t read it here.
It’s because Windows is crap software. Just stop using anything Microsoft makes.
This was very much not caused by windows
In a way, it was. If Windows was not as crappy as it is, external solutions would not be needed.
I don’t hear about billions of Linux or Mac computers going down all at the same time. I’m hearing that windows allows a simple text file change to bring down all of them at the same time.
This happened because a file that CrowdStrike pushed out, which by their own processes is not one that is signed, was immediately pushed out with one of their updates. This update was pushed directly through CrowdStrike’s own method, not via Windows Update. CrowdStrike maintains this capability in order to quickly respond to and prevent security threats. The fact that they have .sys files that aren’t signed is crazy on its own, and a huge screwup by CrowdStrike. So many companies relied upon and trusted this company because up until now, everybody considered it a great product, so it was extremely popular and prevalent. It’s been a huge wake up call for everybody in I.T.
JeeBaiChow@lemmy.world 3 months ago
Whoda thunk automatic updates to critical infrastructure was a good idea?
Toribor@corndog.social 3 months ago
Many compliance frameworks require security utilities to receive automatic updates. It’s pretty essential for effective endpoint protection considering how fast new threats spread.
The problem is not the automated update, it’s why it wasn’t caught in testing and how the update managed to break the entire OS.
jbloggs777@discuss.tchncs.de 3 months ago
It is pretty easy to imagine separate streams of updates that affect each other negatively.
CrowdStrike does its own 0-day updates, Microsoft does its own 0-day updates. There is probably limited if any testing at that critical intersection.
If Microsoft 100% controlled the release stream, otoh, there’d be a much better chance to have caught it. The responsibility would probably lie with MS in such a case.
LainTrain@lemmy.dbzer0.com 3 months ago
Nah EDR is pointless like all of cybersecurity. All these compliance frameworks are just a further grift to get a slice of B2B procurement budgets. The practice of cybersecurity has caused a more severe widespread outage than any malware ever could.
LodeMike@lemmy.today 3 months ago
Hospital stuff was affected. Most engineers are smart enough to not connect critical equipment to the Internet, though.
arunwadhwa@lemmy.world 3 months ago
I’m not in the US, but my other medical peers who are mentioned that EPIC (the software most hospitals use to manage patient records) was not affected, but Dragon (the software by Nuance that we doctors use for dictation so we don’t have to type notes) was down. Someone I know complained that they had to “type notes like a medieval peasant.” But I’m glad that the critical infrastructure was up and running. At my former hospital, we used to always maintain physical records simultaneously for all our current inpatients that only the medical team responsible for those specific patients had access to just to be on the safe side.
Juvyn00b@lemmy.world 3 months ago
I work healthcare adjacent and some providers were affected as expected. Hoping as well that those critical systems were not, but that chance is non zero.