Clownstrike

[Sock Puppet]

Thrad for the ongoing mini-apocalypse.

Originally Posted by JoeP

If you are involved in IT in any way you've probably heard about this massive global outage of Microsoft servers and desktops due to a duff update from security firm Crowdstrike, or Clownstrike as they're now known.

‘Largest IT outage in history’ hits Microsoft Windows and causes global chaos — The Guardian

CrowdStrike fixes start at "reboot up to 15 times" and get more complex from there | Ars Technica

2 of my direct reports (and maybe the 3rd but he's out on jury duty so he may not have tried to log on yet) are out of commission, but mine own self and the rest of the team seem fine. My IT department says they've scrubbed all of the company servers, and are slogging through getting individual equipment back up, but "Please note that, unfortunately, we are finding a handful of computers cannot be recovered and will need replacement."

[Crumb]

Didn't notice it at all at my workplace.

[specious_reasons]

I am mostly working, but this caused a shitstorm on the service I work on.

The public reports seem accurate. Clownstrike pushed an update onto Windows servers in the cloud which blue-screened a lot of cloud compute. They backed out of the update once they discovered the problem, but it's kind of hard to recover once the file causing the BSOD is actually on the computer. If you're lucky, the system will download the patched version before loading the borked one and come up. If you're not, it's not a trivial thing to deal with it.

Most cloud compute updates automatically. Nobody wants to be manually updating thousands of servers, so shit like this is automated. There's supposed to be strategies to roll out updates that minimize the damage if something bad happens. I wonder who pushed a broken update into production systems?

[Ensign Steve]

I found this comment on LinkedIn fairly insightful.

You're right, patches should be tested in a subset of your infrastructure to ensure functionality and security. The problem is that this process is commonly done for Operating Systems and production applications, but tend to exclude endpoint security software. The idea that it is safer to gamble on the integrity of the patch vs the threat of a new exploit.

This event might shatter that construct and force an adjustment to the test plans to include security software to that mentioned 5%.

[Ensign Steve]

At my jerb, managers' laptops are fucked, but engineers are fine because we are on macs. And all our actual services run on non-MS cloud servers, so no issues there. Other than some individuals being mildly inconvenienced, in that they can only access their emails, slack, and google docs via phones, tablets, or a spare uncompromised laptop, you can't even tell there was an incident by looking around.

[specious_reasons]

Endpoint security software is just as vulnerable to bugs as any other software, I don't agree with that priority.

[Ensign Steve]

Yeah I don't necessarily agree either, but I understand the point about the risk trade off.

[vremya]

It didn't affect me either, but there was a big EXPECT DELAYS - CHECK FLIGHTS sign at the airport entrance

[ceptimus]

Some parts of the world more affected than others, because of how the world spins on its axis, and the relative position of the sun. Computers tend to check for, and apply, updates in the wee small hours when they're less busy. The rollout started in Asia and Australia and then moved to Europe. Before America was badly hit, one of the clowns at Clownstrike had noticed something was wrong, and pulled the update.

[Miisa]

I dodged this bullet as CrowdStrike is not in use in my area at all afaik, didn't get a single IT contact about it apart from some users had issues with certain sites and programs being slow etc., and M365 apparently stopped working for a while there. Of course, that was just luck on my part.
Still was counting the minutes until I could clock out and declare myself safe from the looming shitornado as the panic about it had been rising my entire work day at UTC +3.

Also, 90% of users and IT workers alike were on holiday.

[JoeP]

Originally Posted by ceptimus

Some parts of the world more affected than others, because of how the world spins on its axis, and the relative position of the sun.

[JoeP]

Massive insider trading alert for CrowdStrike days before global outage

I'm sure it's a complete coincidence.

[JoeP]

ded68163e370da37.png

[JoeP]

This is pretty representative

[ceptimus]

The CEO of Clownstrike was only paid $38 million last year. Obviously they need to pay more to get a more competent leader.

[Kamilah Hauptmann]

mqbik0yqnkdd1.png