Computers & Hardware

For those sysadmins affected, we wish you well and we hope the overtime pay is great. Luckily the cause is quite well known and fixes are documented. God speed on implementing them!

For those not affected, remember that shit happens. It might not be you today, but it could well be next time. Don't rest on your laurels, make sure you have recovery procedures in place.

For those not sysadmins and are here with popcorn, enjoy the show! This will be going on for many more hours, and probably won't be entirely mitigated until next week.

My company laid off the whole IT team including me about a month ago and outsourced it overseas.

Former coworker just sent me a picture of the HR lady carrying the monitor from her computer to the server room while on the phone with support to try to resolve the crowdstrike outage.

It’s going to be rough for companies with only remote support.

Update: Another former IT coworker reached out to the company and offered to come back and help. They told him “Thanks but we are sure this will be resolved before we could even get you through orientation”.

I think orientation is three days or something if I remember right.

Update 2, the group chat is blowing up haha: CIO just came in and she is flipping out on everyone. She just told my buddy to get dell on the phone right now, lol. HR lady is crying apparently :(

Also they can’t find anybody with keycard access to the second server room and can’t create any new keycards.

Update 3, probably last update: it seems that the CIO just learned that this is a global outage and my buddy said she looks super relieved. All upper leadership went into a closed door meeting. My buddy is still on hold with dell, he works in finance. Everyone else is just sitting around. HR lady went home.

Mini update: Hourly staff sent home but salary staff have to stay. Food is being delivered for the senior leadership meeting but nobody else. My buddy is still on hold with dell.

Resolution update: The CEOs nephew came in because he’s good with computers. He’s going around getting everyone’s workstations back up. My buddy says it looks like he’s following instructions he found on Reddit. Now I’m going to quote the exact description he sent me:

“dude this guy looks like if Timothy chalamet went to the gym six day a week but he’s wearing a shirt with a anime girl that says demon slayer? WTH also the girls in accounting won’t stop talking about how good he smells 🤮”

So dude if you are on here the girls in accounting appreciate your help.

A couple other tidbits: Building maintenance had to come open the server room door.

The CEO screamed at the phone support guys to give his nephew what ever he needed (I’m assuming credentials)

The CIO was heard through the wall defending themselves by saying “I’m not technical, I was brought of for my leadership abilities”

Dominos was delivered for all the staff that had to stay.

Dell never picked up.

Fellow sysadmins,

I am beyond pissed off right now, in fact, I'm furious.

WHY DID CROWDSTRIKE NOT TEST THIS UPDATE?

I'm going onto hour 13 of trying to rip this sys file off a few thousands server. Since Windows will not boot, we are having to mount a windows iso, boot from that, and remediate through cmd prompt.

So far- several thousand Win servers down. Many have lost their assigned drive letter so I am having to manually do that. On some, the system drive is locked and I cannot even see the volume (rarer). Running chkdsk, sfc, etc does not work- shows drive is locked. In these cases we are having to do restores. Even migrating vmdks to a new VM does not fix this issue.

This is an enormous problem that would have EASILY been found through testing. When I see easily -I mean easily. Over 80% of our Windows Servers have BSOD due to Crowdstrike sys file. How does something with this massive of an impact not get caught during testing? And this is only for our servers, the scope on our endpoints is massive as well, but luckily that's a desktop problem.

Lastly, if this issue did not cause Windows to BSOD and it would actually boot into Windows, I could automate. I could easily script and deploy the fix. Most of our environment is VMs (~4k), so I can console to fix....but we do have physical servers all over the state. We are unable to ilo to some of the HPE proliants to resolve the issue through a console. This will require an on-site visit.

Our team will spend 10s of thousands of dollars in overtime, not to mention lost productivity. Just my org will easily lose 200k. And for what? Some ransomware or other incident? NO. Because Crowdstrike cannot even use their test environment properly and rolls out updates that literally break Windows. Unbelieveable

I'm sure I will calm down in a week or so once we are done fixing everything, but man, I will never trust Crowdstrike again. We literally just migrated to it in the last few months. I'm back at it at 7am and will work all weekend. Hopefully tomorrow I can strategize an easier way to do this, but so far, manual intervention on each server is needed. Varying symptom/problems also make it complicated.

For the rest of you dealing with this- Good luck!

*end rant.

Normally the bane of my existence is not having the budget for things like a proper EDR solution. But where are my Defender homies today? Hopefully having a relatively chill Friday?

WHAT ABOUT BITLOCKER?!?!?

Ive answered this 500x in comments...

Can easily be modified to work on bitlocker. WinPE can do it. You just need a way to map the serialnumber to the bitlocker key and unlock it before you delete the file.

wouldnt let me post this, I guess because its too useful.

I fixed the July 19th 2024 issue on 1100 machines in 30 minutes using the following steps.

I modified our standard WinPE image file (from the ADK) to make it delete the file 'C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys' using the following steps.

If you don't already have the appropriate ADK for your environment download it. The only problem with using a bare WinPE image is it may not have the drivers. Another caveat is that this most likely will not work on systems with encrypted filesystems.

Mount the WinPE file with Wimlib or using Microsoft's own tools, although Microsoft's tools are way clunkier and primative.

Edit startnet.cmd and add:

del C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys

exit

to it.

Save startnet.cmd [note the C:\ might be different for you on your systems but it worked fine on all of mine]

Unmount the WinPE image

Copy the WinPE image to either your PXE server or to a USB drive of some kind and make it BOOTABLE using Rufus or whatever you want.

Boot the impacted system.

Hope this helps someone. Would appreciate upvotes because this solution would save people from having to work all weekend and also if it's automatic it's less prone to fat fingering.

Also I am pretty sure that Crowdstrike couldve made this change automatically undoable by just using the WinRE partition.

@tremens suggested that this step might help with bitlocker in WinPE 'manage-bde -unlock X: -recoverypassword <recovery key>' should work in WinPE.

All the corporations that have fired their local IT and offshored talent over the last couple of years so they can pay employees $2 an hour have learned a big lesson today.

Hire quality, local IT.

You are going to need them.