Submitted 8 hours ago by Ek-Hou-Van-Braai@piefed.social to selfhosted@lemmy.world
https://media.piefed.social/posts/Qw/sk/QwskwvFHcPlFovs.pn
All of your systems are set up, but are they capable of being redeployed using a configuration management software package? Ansible or something like that?
Oh. They’re not. Well, that’s probably okay. I mean, you could probably go manually reproduce configurations, more or less.
You should use Arch, then you can update every 15 minutes 🤭
Buy a UPS and setup a NUT server on the spare raspberry pi you have lying around.
I wish it was stable
I had a drive die yesterday
Gotta be honest, my home lab chugs along quite happily.
Atomic fedora makes it hard to break, and then all the services are containerized and managed by configuration and just files only.
When there’s an update to a service: just pull service. Firewall needs configuring: just firewall-reset && just firewall-enable.
The only flaky thing is a vpn that I run through glutan and I’m thinking of dumping that provider.
I had a automatic reboot of all VMs and the hypervisor because of a kernel update at night. Nextcloud decided to start in maintenance mode and Jellyfin refused to start because the cache folder didn’t have enough space left. Authentik also complained about outdated provider configuration…
Need to investigate the Nextcloud and Authentic issue during weekend 🤗
Barring any hardware issues or external factors, will it run for 10000 years? Any logs not properly rotated? And other outputs accumulating and eventually filling up a filesystem?
Man I always get sad when I see this meme format because the story behind it is so fucking tragic… :(
What story?
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
| Fewer Letters | More Letters |
|---|---|
| DNS | Domain Name Service/System |
| HA | Home Assistant automation software |
| ~ | High Availability |
| PSU | Power Supply Unit |
[Thread #161 for this comm, first seen 13th Mar 2026, 11:00] [FAQ] [Full list] [Contact] [Source code]
I havent messed much with my servers in 2 years. I think that means I’ll hit my RIO in another 5 :)
Do your backups work?
Have you tested your backups recently? Having them complete is one thing, having the data you need for recovery is another. Have you backed up your vm configurations and build scripts?
Go test your latest backup!
Restore is future me’s problem. Fuck that guy :D
I haven’t messed with my raspberry pi in maybe a month… And I think one of my backups got corrupted because I receive an email saying that it failed along with tons of errors every night. Hmm, maybe I should get to that soon…
You need monitoring
I’m remembering a very not fun discussion my team had about “the monitoring system not sending any alerts doesn’t inherently mean everything is ok” after an outage that was missed by our monitoring system.
You need to make sure you’re monitoring connectivity as well as specific problem states. No data is a problem state often overlooked, and it’s not always considered for every resource type in these systems out of the box.
And you probably want a heartbeat notification. Yes, it’s noise, but if you don’t see anything from monitoring you need to question if monitoring is the thing that broke. It sending out a notification every so often going “yes I am online” is useful.
One alert daily reporting that there are no alerts is probably good for a home lab…
Kubernetes?
InnerScientist@lemmy.world 4 hours ago
If logging is down and there’s no one around to log it, is it really down?
Agent641@lemmy.world 2 hours ago
Who will log the loggers?