Jerboa crashed mid-comment so i’ll be brief.

Save yourself pain and increase your happiness by

• using btrfs or zfs (snapshots, checksum and self-healing is great)
• using declarative approach rather than imperative, and keep a copy of configs elsewhere (I accidentally nuked my system multiple times, you should expect to do the same)
• keeping backups. If zfs, github.com/jimsalterjrs/sanoid and syncoid are great discourse.practicalzfs.com/t/…/1611
• have an extra tiny machine running the same system and workloads, where you test potentially risky stuff before doing so on the prod server
• metrics solutions like prometheus and grafana are your friend