Comment on Kubernetes storage backends
F04118F@feddit.nl 1 week ago
I tried Longhorn, and ended up concluding that it would not work reliably with Volsync. Volsync (for automatic volume restore on cluster rebuild) is a must for me.
I plan on installing Rook-Ceph. I’m also on 1Gb/s network, so it won’t be fast, but many fellow K8s home opsers are confident it will work.
Rook-ceph does need SSDs with Power Loss Protection (PLP), or it will get extremelly slow (latency). Bandwidth is not as much of an issue. Find some used Samsung PM or SM models, they aren’t expensive.
InnerScientist@lemmy.world 1 week ago
Thanks for the info!
I’ll try Rook-Ceph, Ceph has been recommended quite a lot now, but my nvme drives sadly don’t have PLP. Afaict that should still work because not all nodes will face power loss at the same time.
I’d rather start with the hardware I have and upgrade as necessary, backups are always running for emergency cases and I can’t afford to replace all hard drives.
I’ll join Home Operations once I’ve worked out how to provision my NixOS base system
F04118F@feddit.nl 1 week ago
The problem with non-PLP drives is that Rook-Ceph will insist that its writes get done in a way that is safe wrt power loss.
For regular consumer drives, that means it has to wait for the cache to be flushed, which takes aaaages (milliseconds!!) and that can cause all kinds of issues. PLP drives have a cache that is safe in the event of power loss, and thus Rook-Ceph is happy to write to cache and consider the operation done.
Again, 1Gb network is not a big deal, not using PLP drives could cause issues.
If you don’t need volsync and don’t need ReadWriteMany, just use Longhorn with its builtin backup system and call it a day.