Crowdstrike Nukes Windows Computers Worldwide

[PavelChekov]

The title says it all. I assume a lot of people here are IT staff who are already aware of this and working their asses off to try and fix it. In that spirit, I made this thread to discuss this (never have I been prouder to have a second laptop running unix) and share how they fixed the problem on their systems.

[nullplan]

As a Linux user who is wholeheartedly into software diversity: HAHA! That's what you all get for all running the same messy snake oil as everyone else.

It is funny, you know. The narrative has always been that we need to protect ourselves against the Russian hackers, or else they might just shut down all our systems. And what did shut down our systems for real? American snake oil.

And please, PLEASE, try to recover damages from Crowdstrike or Microsoft (because during the outage, Microsoft 365 was also down). Because otherwise, things like this will just keep happening until everyone's broke.

and share how they fixed the problem on their systems.

Crowdstrike already published a hotfix this morning, according to the radio at least. I wouldn't know. I don't touch "security" snake oil with a ten foot barge pole.

[iansjack]

I’m glad that you think that millions of people being inconvenienced, travel plans disrupted, operations cancelled is funny.

I can tell that you have no experience of running computer operations. These things happen, and can happen to anyone and any system. My sympathies to the sysadmins who have to deal with this outage.

[PavelChekov]

They pushed a patch, but it required booting into safe mode and deleting a file, and I believe people who used Bitlocker with the key stored on a Windows machine were SOL, but I don't know. I share in ianjacks sympathy for the people who are getting it rained down on.

[nullplan]

I’m glad that you think that millions of people being inconvenienced, travel plans disrupted, operations cancelled is funny.

You laugh or you cry. The people that did not decide for this state of affairs to come about have my sympathy, but the deciders absolutely not. They made their bed. Well, they made a bed for you to lie in. Never mind the vipers in there.

I was completely serious when I said that lawsuits need to happen. The same is true when a new ransomware worm drops, and the victim is found to have used Exchange and Active Directory. At this point, that is gross negligence.

These things happen, and can happen to anyone and any system.

But they happen at this scale only because so many people decide to run the same stuff. It wouldn't be nearly as much of a problem if the software causing this was not so widespread. The way the biological world resists attacks is with diversity so that something that kills you may not affect a cockroach, and the technological world needs to do the same. Else one successful attack can shutdown the world. As we've just seen.

BTW, the news announced that it wasn't a cyber attack. How do I put this nicely? Hanlon's razor says to never attribute to malice that which can be explained by incompetence, but I have long held that there comes a point where the difference is moot. I see smouldering ruins where working infrastructure used to be and you tell me this wasn't an attack?

[iansjack]

Don’t be silly. Linux and various Unix variants are widely used. A problem like this could just as easily strike them.

I don’t think you appreciate the efforts, and the strain and stress placed on them, that IT synopsis put in keeping the systems that you take for granted running.

And, no - I don’t have to laugh at people unable to obtain medication and missing out on operations.

[PavelChekov]

https://xkcd.com/2961/

[nullplan]

and I believe people who used Bitlocker with the key stored on a Windows machine were SOL,

Yes, that is the tradeoff with a cryptoroot - if there's something on the file system you need to change, but you can't get at the key, then the very nature of encryption means you can't do it.

This wouldn't be a problem if they'd used Linux with LUKS, because then any Linux boot stick could mount the partition, given the password. But alas...

Don’t be silly. Linux and various Unix variants are widely used. A problem like this could just as easily strike them.

Even just Linux variants are far more diverse that the uniform Windows installations we see. For example, there was that xz hack a couple months ago. It would have had the potential to backdoor all Debian and Red Hat servers, had it not been discovered for a little longer. But for one, it was discovered, and for two, it was merely all Debian and Red Hat servers running OpenSSH. If you use Dropbear, you're not affected. Devuan? Not affected. Buildroot? Not affected. See what I mean? And Debian Stable was never affected for never having an xz version new enough.

If this had been on Windows, there'd be no choice but to use the MS supplied OpenSSH server, pre-backdoored for your convenience.

I don’t think you appreciate the efforts, and the strain and stress placed on them, that IT synopsis put in keeping the systems that you take for granted running.

I do. Believe me, I do. And they are probably underpaid (or at least, when last I looked for a job, sysadmin positions paid far too little), and if they all went on strike tomorrow, which they'd be arguably justified in doing, then a whole lot of things would just stop working. But I can also tell you, they don't care about some pseudonym on the internet laughing. And I don't even laugh at them.

I laugh at the CEOs, chasing their KPIs while blissfully unburdened by any competence. I laugh at the "security" "experts" that sell checklists which tell you to buy more snake oil. I laugh at the snake oil salesmen, that have had their scam catch up to them once more (seriously, why did anyone ever accept kernel module snake oil?). I laugh at the media personalities, ludicrously telling me that this was "not an attack".

You might notice that the laughter is getting progressively more bitter, because we've entrusted these people with our fortunes. This is me dancing on the volcano. This time, I wasn't hurt, and I keep telling people what to do so they probably won't. We'll see what happens.

You seem to misunderstand the saying "you laugh or you cry". Imagine you are on a train. The train is stopped outside a station at a red entry signal, because the switch tower of the station has been digitized, and it is running on a Windows computer that is currently boot looping. Nothing you say or do will change the situation any (unless you do something stupid like pull the emergency brake). Your options are to laugh at the absurdity of the situation, or to despair and throw yourself out the window. As time goes on, people gravitate toward one or the other.

And, no - I don’t have to laugh at people unable to obtain medication and missing out on operations.

I don't laugh at those, either.

[lambduh]

I'm still a bit confused about who this effected. This module isn't included in a vanilla windows installation, right? Does the module get installed as a common dependency to other things people install intentionally?

[iansjack]

The problem is with third-party security software, so it is only organizations that use this software on their computers that are affected. It’s, rather unfairly, been described as a Microsoft or Windows problem. It’s not - it’s a CrowdStrike problem.

Individual users of Windows are not affected.

[chase]

Crowdstrike is very common for enterprise companies.

I avoid restarting my corporate laptop for the last couple of days.

[lambduh]

The problem is with third-party security software, so it is only organizations that use this software on their computers that are affected. It’s, rather unfairly, been described as a Microsoft or Windows problem. It’s not - it’s a CrowdStrike problem.

Individual users of Windows are not affected.

It's a microsoft problem because they have a process to vet drivers and they signed off on this one. There was a lot of nuance lost in the early reporting, and it turns out that crowdstrike does this all the time. But microsoft signed a driver that loads unsigned and unvalidated data files into supervisor mode. That's a microsoft problem.

[iansjack]

It wasn’t the driver that caused the problem but a data file uploaded by that driver. It would be unreasonable, and unworkable, for Microsoft to vet and approve every update file loaded by every security program. CrowdStrike have to accept the responsibility for this.

If I had my car serviced by a garage that was certified by Toyota as an approved dealer and they fitted substandard brake pads, my beef would be with the garage rather than Toyota. (But I would expect, after the event, Toyota to investigate the failure. I’m confident that Microsoft will be having a conversation with CrowdStrike.)

[nullplan]

Well, I stand vindicated in at least one point: CrowdStrike will be liable for damages in France, based on the OVH precedent.

It would be unreasonable, and unworkable, for Microsoft to vet and approve every update file loaded by every security program.

Certainly would. Therefore, it would be important for Microsoft to prevent drivers from loading data files to circumvent Microsoft scrutiny. It is generally good practice to measure and minimize attack surface. Also, code signing isn't worth anything if the signed driver can just load unsigned data that changes the behavior of the signed part.

You know, Linux drivers don't really load data files. Linux drivers have interfaces to change the things that need changing, and userspace programs that interpret data files to set up those things. This is what happens with loadkeys, for example, which loads the keyboard map in use for the virtual terminal.

[iansjack]

Unfortunately, without that facility security software wouldn’t be able to do its job. This was pretty much forced on Microsoft by the EU who wanted to avoid Microsoft’s monopoly on security software for Windows. Microsoft being the only people who could provide APIs giving low-level access is exactly what the EU forbids.

Not allowing companies like CrowdStrike to have low-level access would be a bit like Toyota forbidding anyone but themselves from make brake pads for my car. That really wouldn’t be in my best interests.

You seem to be under the impression that there are no security problems with Linux. That’s a dangerous misconception. As the use of Linux becomes more widespread the threats grow: https://www.sans.org/blog/linux-intrusi ... g-problem/