Homelab upgrade - "Modern" alternatives to NFS, SSHFS?

Submitted ⁨⁨1⁩ ⁨week⁩ ago⁩ by ⁨MajorSauce@sh.itjust.works⁩ to ⁨selfhosted@lemmy.world⁩

Hi all!

I will soon acquire a pretty beefy unit compared to my current setup (3 node server with each 16C, 512G RAM and 32T Storage).

Currently I run TrueNAS and Proxmox on bare metal and most of my storage is made available to apps via SSHFS or NFS.

I recently started looking for “modern” distributed filesystems and found some interesting S3-like/compatible projects.

To name a few:

MinIO
SeaweedFS
Garage
GlusterFS

I like the idea of abstracting the filesystem to allow me to move data around, play with redundancy and balancing, etc.

My most important services are:

Plex (Media management/sharing)
Stash (Like Plex 🙃)
Nextcloud
Caddy with Adguard Home and Unbound DNS
Most of the Arr suite
Git, Wiki, File/Link sharing services

As you can see, a lot of download/streaming/torrenting of files accross services.

Currently the setup is messy due to the organic evolution of my setup, but since I will upgrade on brand new metal, I was looking for suggestions on the pillars.

So far, I am considering installing a Proxmox cluster with the 3 nodes and host VMs for the heavy stuff and a Docker VM.

How do you see the file storage portion? Should I try a full/partial plunge info S3-compatible object storage? What architecture/tech would be interesting to experiment with?

Or should I stick with tried-and-true, boring solutions like NFS Shares?

Thank you for your suggestions!

source

Comments

Sort:hotnew top

just_another_person@lemmy.world ⁨1⁩ ⁨week⁩ ago
“Boring”? I’d be more interested in what works without causing problems. NFS is bulletproof.

source
- 486@lemmy.world ⁨1⁩ ⁨week⁩ ago
  
  NFS is bulletproof.
  
  For it to be bulletproof, it would help if it came with security built in. Kerberos is a complex mess.
  
  source
  - Trincapinones@lemmy.dbzer0.com ⁨1⁩ ⁨week⁩ ago
    Yeah, I’ve ended up setting up VLANS in order to not deal with encryption
    
    source
- MajorSauce@sh.itjust.works ⁨1⁩ ⁨week⁩ ago
  You are 100% right, I meant for the homelab as a whole. I do it for self-hosting purposes, but the journey is a hobby of mine.
  
  So exploring more experimental technologies would be a plus for me.
  
  source
  - just_another_person@lemmy.world ⁨1⁩ ⁨week⁩ ago
    Most of the things you listed require some very specific constraints to even work, let alone work well. If you’re working with just a few machines, no storage array or high bandwidth networking, I’d just stick with NFS.
    
    source
    -> View More Comments
ShortN0te@lemmy.ml ⁨1⁩ ⁨week⁩ ago
sshfs is somewhat unmaintained, only “high-impact issues” are being addressed github.com/libfuse/sshfs

I would go for NFS.

source
- chtk@feddit.nl ⁨1⁩ ⁨week⁩ ago
  And if you need to mount a directory over SSH, I can recommend rclone and its mount subcommand.
  
  source
- Appoxo@lemmy.dbzer0.com ⁨1⁩ ⁨week⁩ ago
  But NFS has mediocre snapshotting capabilities (unless his setup also includes >10g nics)
  
  source
  - ShortN0te@lemmy.ml ⁨1⁩ ⁨week⁩ ago
    I assume you are referring to Filesystem Snapshotting? For what reason do you want to do that on the client and not on the FS host?
    
    source
    -> View More Comments
possiblylinux127@lemmy.zip ⁨1⁩ ⁨week⁩ ago
What’s wrong with NFS? It is performant and simple.

source
- forbiddenlake@lemmy.world ⁨1⁩ ⁨week⁩ ago
  By default, unencrypted, and unauthenticated, and permissions rely on IDs the client can fake.
  
  May or may not be a problem in practice, one should think about their personal threat model.
  
  Mine are read only and unauthenticated because they’re just media files, but I did add unneeded encryption via ktls because it wasn’t too hard to add (I already had a valid certificate to reuse)
  
  source
  - possiblylinux127@lemmy.zip ⁨1⁩ ⁨week⁩ ago
    NFS is good for hypervisor level storage. If someone compromises the host system you are in trouble.
    
    source
    -> View More Comments
- vext01@lemmy.sdf.org ⁨1⁩ ⁨week⁩ ago
  NFS is fine if you can lock it down at the network level, but otherwise it’s Not For Security.
  
  source
  - Appoxo@lemmy.dbzer0.com ⁨1⁩ ⁨week⁩ ago
    NFS + Kerberos?
    
    But everything I read about NFS amd so on: You deploy it on a dedicated storage LAN and not in your usual networking LAN.
    
    source
    -> View More Comments
- linuxguy@lemmy.gregw.us ⁨1⁩ ⁨week⁩ ago
  Gotta agree. Even better if backed by zfs.
  
  source
scumola@sh.itjust.works ⁨1⁩ ⁨week⁩ ago
I’d only use sshfs if there’s no other alternative. Like if you had to copy over a slow internet link and sync wasn’t available.

NFS is fine for local network filesystems. I use it everywhere and it’s great. Learn to use autos and NFS is just automatic everywhere you need it.

source
- azron@lemmy.ml ⁨1⁩ ⁨week⁩ ago
  *autofs
  
  source
non_burglar@lemmy.world ⁨1⁩ ⁨week⁩ ago
Your workload just won’t see much difference with any of them, so take your pick.

NFS is old, but if you add security constraints, it works really well. If you want to tune for bandwidth, try iSCSI , bonus points if you get zfs-over-iSCSI working with tuned block size. This last one is blazing fast if you have zfs at each and you do Zfs snapshots.

Beyond that, you’re getting into very tuned SAN things, which people build their careers on, its a real rabbit hole.

source
- possiblylinux127@lemmy.zip ⁨1⁩ ⁨week⁩ ago
  NFS with security does harm performance. For raw throughput it is best to use no encryption. Instead, use physical security.
  
  source
  - non_burglar@lemmy.world ⁨1⁩ ⁨week⁩ ago
    I don’t know what you’re on about, I’m talking about segregating with vlans and firewall.
    
    If you’re encrypting your San connection, your architecture is wrong.
    
    source
    -> View More Comments
nesc@lemmy.cafe ⁨1⁩ ⁨week⁩ ago
Gluster is ~~shit~~ really bad, garage and minio are great. If you want something tested and insanely powerful go with ceph, it has everything. Garage is fine for smaller installations, and it’s very new and not that stable yet.

source
- possiblylinux127@lemmy.zip ⁨1⁩ ⁨week⁩ ago
  Ceph isn’t something you want to jump into without research
  
  source
- corsicanguppy@lemmy.ca ⁨1⁩ ⁨week⁩ ago
  
  go with ceph[:] it has everything
  
  I heard running an object store as a filesystem was considered risky, but that’s not why it sometimes hoses your storage.
  
  source
  - nesc@lemmy.cafe ⁨1⁩ ⁨week⁩ ago
    Last time I had a problem with ceph losing data was during 0.10, does it still happen?
    
    source
- MajorSauce@sh.itjust.works ⁨1⁩ ⁨week⁩ ago
  Darn, Garage is the only one that I successfully deployed a test cluster.
  
  I will dive more carefully into Ceph, the documentation is a bit heavy, but if the effort is worth it…
  
  Thanks.
  
  source
  - nesc@lemmy.cafe ⁨1⁩ ⁨week⁩ ago
    I had great experience with garage at first, but it crapped itself after a month, it was like half a year ago and the problem was fixed, still left me with a bit of anxiety.
    
    source
    -> View More Comments
PunkiBas@lemmy.world ⁨1⁩ ⁨week⁩ ago
I’m using ceph on my proxmox cluster but only for the server data, all my jellyfin media goes into a separate NAS using NFS as it doesn’t really need the high availability and everything else that comes with ceph.

It’s been working great, You can set everything up through the Proxmox GUI and it’ll show up as any other storage for the VMs. You need enterprise grade NVMEs for it though or it’ll chew through them in no time. Also a separate network connection for ceph traffic if you’re moving a lot of data.

Very happy with this setup.

source
LiPoly@lemmynsfw.com ⁨1⁩ ⁨week⁩ ago
If you want to try something that’s quite new and mostly unexplored, look into NVMe over TCP. I really like the concept, but it appears to be too new to be production ready. Might be a good fit for your adventurous endeavors.

source
- gloriousspearfish@feddit.dk ⁨1⁩ ⁨week⁩ ago
  This is just block device over network, it will not allow the use cases OP is asking for. You will still need a filesystem and a file-serving service on top of that.
  
  source
  - LiPoly@lemmynsfw.com ⁨1⁩ ⁨week⁩ ago
    I agree, but it’s clear that OP doesn’t want a real solution, because those apparently are boring. Instead, they want to try something new. NVMe/TCP is something new. And it still allows for having VMs on one system and storage on another, so it’s not entirely off topic.
    
    source
naomi@lemmy.amethyst.name ⁨1⁩ ⁨week⁩ ago
I use Ceph/CephFS myself for my own 671TiB array (382TiB raw used, 252TiB-ish data stored) – I find it a much more robust and better architected solution than Gluster. It supports distributed block devices (RBD), filesystems (CephFS), and object storage (RGW). NFS is pretty solid though for basic remote mounting filesystems.

source
ShellMonkey@lemmy.socdojo.com ⁨1⁩ ⁨week⁩ ago
I’ve used MinIO as the object store on both Lemmy and Mastodon, and in retrospect I wonder why. Unless you have clustered servers and a lot of data to move it’s really just adding complexity for the sake of complexity. I find that the bigger gains come from things like creating bonded network channels and sorting out a good balance in the disk layout to keep your I/O in check.

source
- Xanza@lemm.ee ⁨1⁩ ⁨week⁩ ago
  I preach this to people everywhere I go and seldom do they listen. There’s no reason for object storage for a non-enterprise environment. Using it in homelabs is just…mostly insane…
  
  source
  - ShellMonkey@lemmy.socdojo.com ⁨1⁩ ⁨week⁩ ago
    Generally yes, but it can be useful as a learning thing. A lot of my homelab use is for purposes of practicing with different techs in a setting where if it melts down it’s just your stuff. At work they tend to take offense of you break prod.
    
    source
lambalicious@lemmy.sdf.org ⁨1⁩ ⁨week⁩ ago
Fam, the modern alternative to SSHFS is literally SSHFS.

All that said, if your use case is mostly downloading and uploading files but not moving them between remotes, then overlaying webdav on whatever you feel comfy on (and that’s already what eg.: Nexctloud does, IIRC) should serve well.

source
catloaf@lemm.ee ⁨1⁩ ⁨week⁩ ago
What are you hosting the storage on? Are you providing this storage to apps, containers, VMs, proxmox, your desktop/laptop/phone?

source
- MajorSauce@sh.itjust.works ⁨1⁩ ⁨week⁩ ago
  Currently, most of the data in on a bare-metal TrueNAS.
  
  Since the nodes will come with each 32TB of storage, this would be plenty for the foreseeable future (currently only using 20TB across everything).
  
  The data should be available to Proxmox VMs (for their disk images) and selfhosted apps (mainly Nextcloud and Arr apps).
  
  A bonus would be to have a quick/easy way to “mount” some volume to a Linux Desktop to do some file management.
  
  source
  - catloaf@lemm.ee ⁨1⁩ ⁨week⁩ ago
    Proxmox supports ceph natively, and you can mount it from a workstation too, I think. I assume it operates in a shared mode, unlike iscsi.
    
    If the apps are running on a VM in proxmox, then the underlying storage doesn’t matter to them.
    
    NFS is probably the most mature option, but I don’t know if proxmox officially supports it.
    
    source
    -> View More Comments
31337@sh.itjust.works ⁨1⁩ ⁨week⁩ ago
NFS gives me the best performance. I’ve tried GlusterFS (not at home, for work), and it was kind of a pain to set up and maintain.

source
jeena@piefed.jeena.net ⁨1⁩ ⁨week⁩ ago
I think you will need to have a mix, not everything is S3 compatible.

But I also like S3 quite a lot.

source
- MajorSauce@sh.itjust.works ⁨1⁩ ⁨week⁩ ago
  I think I am on the same page.
  
  I will provably keep Plex/Stash out of S3, but Nextckoud could be worth it? (1TB with lots of documents and medias).
  
  How would you go for Plex/Stash storage?
  
  Keeping it as a LVM in Proxmox?
  
  source