Help me figure out the way to scale laterally?

Submitted ⁨⁨1⁩ ⁨day⁩ ago⁩ by ⁨EpicFailGuy@lemmy.world⁩ to ⁨selfhosted@lemmy.world⁩

https://lemmy.world/pictrs/image/bef53e64-5975-4605-bed9-88d6ba5fef3f.png

I have a home lab I use for learning and to self host a couple of services for me and my extended family.

Nextcloud instance with about 1TB
Couple of websites
Couple of game servers

I’m running off an R430 with twin E5-2620 V3s 128GB and spinning rust storage.

When I deployed NC I did not think it thru, and I stored all the data locally, which causes the instance to be too big to backup normally.

As a solution, I’ve split the NC software into it’s own LXC and a NAS into another and I’m thinking about hosting a cheap NUC NAS to rsync the files.

I would also like to distribute the load of my server into separate nodes so I can get more familiar with HA and possibly hyper converged infrastructure.

I would also like to have wo nodes locally to be able to work on one without bringing down services.

Any advice / tips?

Should I skip the NAS and go straight into Ceph?

Would 3x NUCs with Intel i5 or i7ths and 32Gb Ram be enough?

Storage wise I’m considering an NVMe to Sata adapter like this [](

Image

) and a mixtures of SSDs and spinning rust.

Otherwise would I be better off with SFF?

I’m looking at cost per watt (current server runs at 168w 90% of the time, looks like those tiny NUCs run about 25W or so and the SFF 50-75W depending on what they have.

I also have a 12U rack at home and I would very much like to keep things racked and neat. It seems a lot easier to rack the NUCs than it would be to do with SFF cases.

Obviously I’m OK with buying new hardware (I’ll be selling the current one once I migrate) that’s part of the “learning” experience.

Otherwise I was considering a single 24 bay disk array with an LSI card in IT mode, but I’m inexperienced with those and I’m not sure about power usage / noise. (the rack does sit next to my workstation)

Plus, most likely I would not expand the storage past 5 or 10 TB on each node.

Any advice or experience you can share would be highly appreciated.

Thanks /c/selfhosted

source

Comments

Sort:hotnew top

ragingHungryPanda@lemmy.zip ⁨1⁩ ⁨day⁩ ago
Id say don’t go down the distributed software route just yet. An option is to keep the NAS and set up a new computer with your router or whatever handling dns to it for services. You should be able to use NAS storage over the network. It’ll be slower, but let’s say you put your database on the big one, then it shouldn’t be so bad since you’ll just make database calls to it.

There are a lot of options. I’m currently learning kubernetes and all that stuff is a big learning curve, so avoid that stuff.

My thoughts: you can do some simple dns load balancing between the two servers at whatever level is handling that. Set up a database on the storage server and let the new computer communicate to it over the network.

source
- EpicFailGuy@lemmy.world ⁨1⁩ ⁨day⁩ ago
  Appreciate your input. K8 is on the roadmap. Currently on portrait using pangolin as a tunneled proxy.
  
  Eventually plan to migrate from Joomla in LXC to a docker swarm load balanced by pangolin.
  
  source
  - ragingHungryPanda@lemmy.zip ⁨1⁩ ⁨day⁩ ago
    It sounds like you’re diving right in. If you go that route, one thing I found was that Ceph wants to work with while drives only, not partitions. I switched to using longhorn so I could take advantage of all of my storage without throwing any away
    
    source
Carrot@lemmy.today ⁨1⁩ ⁨day⁩ ago
I subscribe to the philosophy that each server should handle one thing. I’ve got a NAS that stores all data for all other servers other than boot drives. I’m using only spinning rust for data, so the network speed is never the bottleneck for my system. My NAS is a 24 bay chassis with the LSI card in IT mode. I got an LSI card that is powered from the PCIe port itself, and power usage for the card itself seems negligible, but spinning 24 drives takes a decent bit of power. I’ve got 20 drives in it now and it’s pretty loud, but substantially quieter than the dell r720 it replaced. It’s in my basement so it doesn’t bother me, but if sound is an issue, and you don’t need a ton of space, definitely go with SSDs. I’ve also got a media server that handles all media streaming (movies/TV/audiobooks/music/ebooks/comics/manga/roms). It reads/writes it’s data to the NAS. I’ve got another server running my personal cloud (nextcloud, password manager, testing new SH services). Again, the nextcloud data is on the NAS. Both servers store backups to the NAS, as well as a second local drive. I’ve also got a handful of raspberry pis running the smart house stuff, and one running the Ubiquiti Controller. All are running the PoE hat with the m.2 port on them for stable boot drives, and store their backups on the NAS. I’ve kind of stopped running proxmox + virtualization when I switched off of the r720, as I find that running Debian on bare metal with btrfs backups is simpler for me, and I run almost all of my services in Docker. I’ve had a motherboard go out on my media server, and was able to swap the motherboard and get everything back up and running in a little under 2 hours. Longest part was the motherboard swap itself.

source
- EpicFailGuy@lemmy.world ⁨1⁩ ⁨day⁩ ago
  Interesting, thanks for sharing.
  
  Any clue what the power draw on the disk array is? I did some basic measurement with the kill-a-watt and a spinner takes about 6-7W where as an SSD takes about 2, the price difference is too much for my use case tho, performance per watt per TB, I’m better off with 1 single disk (or a mirror pair) of 6 TB in spinning rust.
  
  I’m not particularly concerned about data security since I’m syncing evrything 3 ways. Whenever one of the drive fails I’ll consider it a “surprise disaster recovery exercise” XD
  
  source
  - Carrot@lemmy.today ⁨1⁩ ⁨day⁩ ago
    Fair, I’ll try to get my kill-a-watt plugged in to check next time the server powers down and report back. Power is fairly cheap where I live, and I’ve got solar, so that’s never been a huge concern for me. I’d have to check, but I’ve always assumed it’s pulling ~10 watts per drive at normal times, and as far as I know my power bill is pretty much reflecting this. (Due to how my data pool works all drives need to be spinning when in use, and my drives get basically zero down time).
    
    And that’s good, when I first got into self hosting I was greedy for storage and didn’t have the money to pay for redundancy, and I got bit a few times. Now my media server is running on 2 8-drive pools, each with two drives of parity. Ends up being around 200TB of useable space. I don’t have backups on my media pools, as right now I’m using 24TB drives and the cost to back that up just doesn’t make sense. I do however have my personal cloud on mirrored drives with a backup at my brother’s house, also on mirrored drives, so it’d be pretty unlikely for me to lose the important stuff.
    
    source
punkibas@lemmy.zip ⁨1⁩ ⁨day⁩ ago
I went with three lenovo tiny 1L for my cluster and am very happy with the result. This thread has a lot of info on their upgradability. I chose them because of the low power consumption and the pci-e slot, planning on upgrading them to 10gbe when I need to, but right now they can idle at as low as 5w.

I’m running a proxmox cluster on them with a dedicated nvme on each one for ceph storage (they have 2 nvmes each) and they work very well, much better in fact than the proxmox cluster with ceph we have at my job that’s using ssds for storage.

Setup was a breeze as proxmox will set it all up from it’s graphical interface automatically and am running all my workloads on VMs and LXCs. HA is also pretty easy with proxmox, and I’ve mingled with both docker swarm and kubernetes by creating VMs on each of my nodes. But ultimately I decided I like the versatility of proxmox with it’s LXCs and VMs better than having to deal with the overhead of both of those solutions. In any case, all those unused VMs are still sitting there, powered off, in case I ever have something I want to manage through swarm or k8s.

It all gets backed up to a proxmox backup server I have set up on another 1L PC with a 5TB drive in it.

Besides that, I have a NAS for my movies, photos, etc. That also gets a copy of the backups and it itself backs up everything important to a remote hetzner storage box with backrest.

I have also set up a VPS with Pangolin for external access that’s working great.

With this setup I can pretty much set up anything I want with HA and know that both me and my friends and family can count on that service to work 100%

source