If it’s stable, it’s not a lab.
That’s infrastructure.
Submitted 7 hours ago by Ek-Hou-Van-Braai@piefed.social to selfhosted@lemmy.world
https://media.piefed.social/posts/Qw/sk/QwskwvFHcPlFovs.pn
Comments
nonentity@sh.itjust.works 3 hours ago
cenzorrll@piefed.ca 1 hour ago
I’ve moved my homelab twice because it became stable, I really liked the services it was running, and I didn’t want to disturb the last lab*cough*prod server.
My current homelab will be moar containers. I’m sure I’ll push it to prod instead of changing the IP address and swapping name tags this time.
squirrel@piefed.zip 7 hours ago
Let’s tinker around and accidentally break something.
wersooth@lemmy.ml 6 hours ago
and debug it until you have to reinstall your entire stack from scarch
SpikesOtherDog@ani.social 5 hours ago
GET OUT OF MY HOUSE!
rosco385@lemmy.wtf 4 hours ago
Are you implying it’s possible to debug without having to reinstall from scratch? Preposterous! 😂
cenzorrll@piefed.ca 1 hour ago
“Damn, I’ve got this Debian server shit down. I wonder how an opensuse server would work out” *installs tumbleweed *
True story
Coleslaw4145@lemmy.world 2 hours ago
No try migrating all your docker containers to podman.
fossilesque@mander.xyz 2 hours ago
Don’t encourage me.
epicshepich@programming.dev 1 hour ago
And then try turning on SELinux!
truthfultemporarily@feddit.org 5 hours ago
Have you tried introducing unnecessary complexity?
Sabata11792@ani.social 4 hours ago
If you know how your setup works, then that’s a great time for another project that breaks everything.
cenzorrll@piefed.ca 1 hour ago
Saturday morning: “Incus and podman seem interesting. I bet I could swap everything over while the family is out this afternoon”
Sunday evening: “Dad, when will the lights work again?”
non_burglar@lemmy.world 2 hours ago
Haha too right mate
InnerScientist@lemmy.world 3 hours ago
Infrastructure diagram? No! In this homelab we refer to the infrastructure hyperdodecahedron.
tal@lemmy.today 19 minutes ago
It seems like a good time to learn graphviz’s dot format for the network layout diagrams, with automated layout.
Avicenna@programming.dev 1 hour ago
You can always configure your vim further
DownByLaw@sh.itjust.works 2 hours ago
Have you already tried implementing an identity provider like Authentik, so you can add OIDC and ldap for all your services, while you are the only one that’s using them? 🤔
tal@lemmy.today 23 minutes ago
Probably a good idea to switch over to WPA-Enterprise using Authentik’s RADIUS server support and let all of the users of your wireless access point log in with their own network credentials, while you’re at it.
epicshepich@programming.dev 1 hour ago
Hey my wife uses some of them too!
PumpkinEscobar@lemmy.world 2 hours ago
Behind a traefik reverse proxy with lets encrypt for ssl even though the services aren’t exposed to the internet?
DownByLaw@sh.itjust.works 1 hour ago
Don’t forget about Anubis and crowdsec to make it even safer inside your LAN
diablomnky666@lemmy.wtf 1 hour ago
To be fair a lot of apps don’t handle custom CAs like they should. Looking at you Home Assistant! 😠
nucleative@lemmy.world 2 hours ago
Never run:
docker compose down && docker compose up -d ``` Right before the end of your day. Ask me how I know 😂
shym3q@programming.dev 1 hour ago
compose upwill automatically recreate with newer images if the new one were pulled. so there is no need forcompose downbtw
Prunebutt@slrpnk.net 6 hours ago
When’s the last time you checked if your backup solution works?
Ek-Hou-Van-Braai@piefed.social 4 hours ago
But if my backups actually work then I miss out on the joy of rebuilding everything from scratch and explaining to my wife why non of the lights in the house work anymore.
piranhaconda@mander.xyz 1 hour ago
What’s a backup solution…? (I’m only being half sarcastic, I really need to set one up, but it’s not as “fun” as the rest of my homelab, send suggestions)
JetpackJackson@feddit.org 6 hours ago
Yesterday! Switched my media server from freebsd to alpine and got the arr stack all set up using the backup zip files
halcyoncmdr@piefed.social 5 hours ago
Backup? Psh… That’s what the lab is for.
FauxLiving@lemmy.world 3 hours ago
The comments in this thread have collectively created thousands of person-hours worth of work for us all…
tal@lemmy.today 6 hours ago
You have remote power management set up for the systems in your homelab, right? A server set up that you can reach to power-cycle other servers, so that if they wedge in some unusable state and you can’t be physically there, you can still reboot them? A managed/smart PDU or something like that? Something like one of these guys?
Oh. You don’t. Well, that’s probably okay. I mean, nothing will probably go wrong and render a device in need of being forcibly rebooted when you’re physically away from home.
sytone@lemmy.world 1 hour ago
Tal just got the chaotic evil tag today.
FauxLiving@lemmy.world 3 hours ago
Oh. You don’t. Well, that’s probably okay. I mean, nothing will probably go wrong and render a device in need of being forcibly rebooted when you’re physically away from home.
*furiously adds a new item to the TODO list*
lemming741@lemmy.world 6 hours ago
Does a $12 Shelly plug count?
TerHu@lemmy.dbzer0.com 5 hours ago
if you can cycle your home assistant with the shelly plug whilst your home assistant is down, yes. from experience it’s really quite annoying to have a smart plug switch off HA…
tychosmoose@lemmy.world 5 hours ago
If you do have the smart PSU and power management server you probably also went down the rabbit hole of scripting the power cycling, right? Maybe made that server hardened against power loss disk corruption so it can be run until UPS battery exhaustion.
What if there is a power outage and NUT shuts everything down? Would be nice to have everything brought back up in an orderly way when power returns. Without manual intervention. But keeping you informed via logging and push notifications.
exu@feditown.com 2 hours ago
I test in my Homeproduction
tal@lemmy.today 6 hours ago
logging is probably down
You do, of course have a dedicated rsyslogd server? An isolated system to which logs are sent, so that if someone compromises your other systems, they can’t wipe traces of that compromise from those systems?
Oh. You don’t. Well, that’s okay. Not every lab can be complete. That Raspberry Pi over there in the corner isn’t actually doing anything, but it’s probably happy where it is. You know, being off, not doing anything.
cenzorrll@piefed.ca 1 hour ago
Hmmm. My pi{VPN,hole,dhcp,HA} has a little bit of overhead left…
probable_possum@leminal.space 6 hours ago
Ah. The approach that squirrel@piefed.zip suggested. ;)
Thanks for the tutorial though.
tal@lemmy.today 6 hours ago
All of those systems in your homelab…they aren’t all pulling down their updates multiple times over your network link, right? You’re making use of a network-wide cache? For Debian-family systems, something like Apt-Cacher NG?
Oh. You’re not. Well, that’s probably okay. I mean, not everyone can have their environment optimized to minimize network traffic.
panda_abyss@lemmy.ca 3 hours ago
You can forgejo with a container index enabled, I don’t know if there’s a way to use that as a proxy for downloading containers though.
the_tab_key@lemmy.world 4 hours ago
I set this up years ago, but then decided it was better to just install different distros on each of my computers. Problem solved?
mech@feddit.org 2 hours ago
Time to switch distros!
Fedegenerate@fedinsfw.app 4 hours ago
Going into spring/summer that’s ideal, I wanna go places do things. Mid winter, I’m feature creeping till something breaks.
AnUnusualRelic@lemmy.world 2 hours ago
That’s not a homelab, that’s a home server.
chaotic_ugly@lemmy.zip 2 hours ago
greedytacothief@lemmy.dbzer0.com 3 hours ago
Yeah, my home server was being a little too stable and I wasn’t really learning anything. So I switched from fedora to proxmox, now I’ve got a nixos vm I’m going to try to get all my services running in.
InnerScientist@lemmy.world 3 hours ago
If logging is down and there’s no one around to log it, is it really down?
Agent641@lemmy.world 1 hour ago
Who will log the loggers?
tal@lemmy.today 6 hours ago
You have an intrusion detection system set up, right? A server watching your network’s traffic, looking for signs that systems on your network have been compromised, and to warn you?
Oh. You don’t. Well, that’s probably okay. I mean, probably nothing on your network has been compromised. And probably nothing in the future will be.
AkatsukiLevi@lemmy.world 4 hours ago
Do you have a spinning fish display in front of your homelab server, right? We all know the spinning fish improves performance and security, it is a indispensable part of homelabbing
Skullgrid@lemmy.world 3 hours ago
J O E L
tal@lemmy.today 5 hours ago
You have all your devices attached to a console server with a serial port console set up on the serial port, and if they support accessing the BIOS via a serial console, that enabled so that you can access that remotely, right? Either a dedicated hardware console server, or some server on your network with a multiport serial card or a USB to multiport serial adapter or something like that, right? So that if networking fails on one of those other devices, you can fire up
minicomor similar on the serial console server and get into the device and fix whatever’s broken?Oh, you don’t. Well, that’s probably okay. I mean, you probably won’t lose networking on those devices.
EonNShadow@pawb.social 2 hours ago
I wish it was stable
I had a drive die yesterday
Admax@lemmy.world 6 hours ago
Then it turns out your monitoring system failed and FUCK IT’S BEEN A MONTH SINCE THE LAST PROPER BACKUP
wizardbeard@lemmy.dbzer0.com 2 hours ago
Hearbeat notifications man. “Yes I am online” email once a day or so. Yeah it’s more emails to delete but it can be a lifesaver.
tal@lemmy.today 6 hours ago
You have squid or some other forward http proxy set up to share a cache among all the devices on your network set up to access the Web, to minimize duplicate traffic?
And you have a shared caching DNS server set up locally, something like BIND?
Oh. You don’t. Well, that’s probably okay. I mean, it probably doesn’t matter that your devices are pulling duplicate copies of data down. Not everyone can have a network that minimizes latency and avoids inefficiency across devices.
InnerScientist@lemmy.world 3 hours ago
That won’t work in most cases, all https traffic isn’t cached unless you mitm https which is a bad idea and not worth it.
Only cache updates those are worth it and most have a caching server option.
tal@lemmy.today 6 hours ago
All of your systems are set up, but are they capable of being redeployed using a configuration management software package? Ansible or something like that?
Oh. They’re not. Well, that’s probably okay. I mean, you could probably go manually reproduce configurations, more or less.
Petter1@discuss.tchncs.de 6 hours ago
You should use Arch, then you can update every 15 minutes 🤭
panda_abyss@lemmy.ca 3 hours ago
Gotta be honest, my home lab chugs along quite happily.
Atomic fedora makes it hard to break, and then all the services are containerized and managed by configuration and just files only.
When there’s an update to a service:
just pull service. Firewall needs configuring:just firewall-reset && just firewall-enable.The only flaky thing is a vpn that I run through glutan and I’m thinking of dumping that provider.
Egonallanon@feddit.uk 6 hours ago
Buy a UPS and setup a NUT server on the spare raspberry pi you have lying around.
PHLAK@lemmy.world 3 minutes ago
Time to start documenting it!
BuboScandiacus@mander.xyz 2 minutes ago
NEVER1!!!11!!