Comment on Kubernetes? docker-compose? How should I organize my container services in 2024?
johntash@eviltoast.org 8 months agoHey, your stack is pretty similar to mine. One thing I recently started testing is Seaweedfs. I saw it listed in your repo too, how are you liking it so far? And do you use it on all of your nodes?
nico@r.dcotta.eu 8 months ago
I struggled a bit to get it up and running well, but now I am happy with it. It’s not too hard to deploy (at least easier than the alternatives), it has CSI which for me was big, and it has erasure coding. The dev that maintains it (yes, the one dev) is very responsive.
It has trade offs, so depending on your needs, I recommend it. Backing store for stateful workloads like postgres DBs? Absolutely not. Large S3 store (with an option for filesystem mount) for storing lots of files? Yes! In that regard it’s good for stuff like Lemmy’s pictrs or immich. I use it as my own Google drive. You can easily replicate in your own cluster, or back it up to an external cloud provider. You can mount it via FUSE on your personal machine too.
Feel free to browse through my setup - if you have specific questions I am happy to answer them.
johntash@eviltoast.org 8 months ago
Thanks! I’ll do some testing over the weekend and see how it goes.
While I’d love to be able to use it for postgres, I figured that wouldn’t work out well so probably won’t try it any time soon. I do have several apps that use sqlite databases though, do you think those would have any issues? e.g. trilium, ntfy, ghost
The main downside to most of the distributed/clustered storage that I’ve tried is they always seem to corrupt sqlite db files due to not supporting locking or some other posix feature. Reading through some older github issues, it looks like that is something the dev of seaweedfs fixed hopefully.
nico@r.dcotta.eu 8 months ago
The problem with using seaweedfs to a back your DBs is more on the filesystem than the implementations of POSIX features. When you are writing to a file, and the connection to seaweedfs breaks (container restart, wifi, you name it), then you might end up with a half-written file. If you upload pictures, this is unlikely, but DBs are doing several writes per second usually. So it is more likely one of those gets interrupted. In my case, my grafana sqlite DB would get corrupted every other week.
What I recommend is using DBs natively in your node’s filesystem, and backing them up to seaweedfs periodically instead. That way your DBs ‘work’ but you can get them running again, and the backup is replicated in the distributed filesystem.
nopersonalspace@lemmy.world 8 months ago
That’s an interesting issue. Do you think the problem would be the same for any CSI plugin? I’m thinking of using my NAS as the storage brains of the operation and hooking it up with NFS or something, but would that have issues with stateful stuff like DB’s too?
johntash@eviltoast.org 8 months ago
What I do right now is I have a rclone sidecar container that uploads files in a directory every few seconds, and I also have another init sidecar that runs before the main application and downloads those files (incl sqlite dbs) to the normal disk. This works okay but feels pretty clunky and can still result in stuff getting corrupted because I’m just backing up the db files and not using any sqlite commands to actually back up the db to another file that isn’t in-use first.
How do you handle a job going from one nomad node to another? Or do you pin jobs like grafana to specific hosts?