Comment

Comment on An angry admin shares the CrowdStrike outage experience

This is why every machine I manage has a second boot option to download a small recovery image off the Internet and phone home with a shell. And a copy of it on a cheap USB stick.

You’re fucking killing it. Stay awesome.

Also gist this up pls. Thanks.

source

Sort:hotnew top

Max_P@lemmy.max-p.me ⁨1⁩ ⁨month⁩ ago
I wish it was more shareable, but it’s also not as magic as it sounds.

Fundamentally it’s just a Linux install with some heavy customizations so that it does one thing only: boot Linux, and just enough prompts to get it online so that the VPN works, and download the root image into RAM that it boots into so I can SSH into the box, and then a bunch of Linux tools for me to use so I can reimage from there, or run a QEMU with the physical disk passed through so I can VNC into an install even if it BSOD.

It’s a Linux UKI (combined kernel+initramfs into a simple EFI file the firmware can boot directly without a bootloader), but you can just as easily get away with a hidden Debian install or whatever. Can even be a second Windows install if that’s your thing. The reason I went this particular route is I don’t have to update it since it downloads it on the fly, much like the Mac recovery. And it runs entirely in RAM afrerwards so I can safely do whatever is needed with the disk.

source
- flop_leash_973@lemmy.world ⁨1⁩ ⁨month⁩ ago
  I dream of working somewhere where this kind of effort is appreciated enough to motivate me to put in the effort of actually doing it.
  
  source
  - Max_P@lemmy.max-p.me ⁨1⁩ ⁨month⁩ ago
    I wish too, it’s only deployed for family and family businesses because I’m a couple thousand miles away from them. I cobbled this together for the explicit purpose of being able to reinstall Windows remotely. It works wonderfully though!
    
    My real job is DevOps and 100% Linux, and most of the cloud servers are disposable and can be simply be rebuilt at the push of a button in some dashboard.
    
    source
- Empricorn@feddit.nl ⁨1⁩ ⁨month⁩ ago
  Yeah, that sounds even more magic! I hope you’re appreciated…
  
  source
  - AlbinoPython@lemmy.world ⁨1⁩ ⁨month⁩ ago
    “It’s not as magic as it sounds”, he says as my eyes gloss over on the first sentence.
    
    source
- HoornseBakfiets@feddit.nl ⁨1⁩ ⁨month⁩ ago
  I wonder what your reasoning was for rolling your own compared to just using something like OpenSuse’s MicroOS
  
  source