Comment on An angry admin shares the CrowdStrike outage experience
Boozilla@lemmy.world 4 months ago
If you have EC2 instances running Windows on AWS, here is a trick that works in many (not all) cases. It has recovered a few instances for us:
- Shut down the affected instance.
- Detach the boot volume.
- Move the boot volume to a working instance in the same region (us-east-1a or whatever).
- Remove the file(s) recommended by Crowdstrike
- Detach and move the volume back over to original instance
- Boot original instance
Alternatively, you can restore from a snapshot prior to when the bad update went out from Crowdstrike. But that is not always ideal.
Defaced@lemmy.world 4 months ago
A word of caution, I’ve done this over a dozen times today and I did have one server where the bootloader was wiped after I attached it to another EC2. Always make a snapshot before doing the work just in case.
Boozilla@lemmy.world 4 months ago
Good advice!