Feasibility & sanity check: Scaling a P330 Tiny to a 3-node Proxmox cluster

Submitted ⁨⁨13⁩ ⁨hours⁩ ago⁩ by ⁨buildmylab@lemmus.org⁩ to ⁨selfhosted@lemmy.world⁩

I’m currently running Proxmox on a single Lenovo ThinkStation P330 Tiny. As my VMs workloads have grown, the 64GB of RAM is consistently sitting near capacity.

To spread the memory load and get proper HA, seamless VMs migration, and easier node maintenance, I’m considering adding two more matching units to build a 3-node cluster.

Planned Per-Node Specs:

. Chassis: Lenovo P330 Tiny . CPU: Intel Core i9-9900T ES . RAM: 64GB DDR4-3200 (running at 2666MHz platform limit) . Boot/Local Storage: 1TB NVMe

Networking: Upgraded to 2.5GbE

Shared Storage:

Synology NAS via iSCSI (SSD cache enabled) over 2.5GbE

Does this setup still offer good value for money, or are there better alternatives in this form factor?

I’d especially appreciate feedback on:

9900T ES stability: Long-term reliability and virtualization quirks under Proxmox.
2.5GbE bandwidth: Is it sufficient for 3 nodes sharing iSCSI, especially during backups or migrations?
Synology iSCSI + SSD cache: Practical experience with this as primary VM shared storage across multiple P330 tiny nodes.
Any other bottlenecks, hidden traps, or limitations I might be overlooking?

original

Comments

Sort:hotnew top

frongt@lemmy.zip ⁨13⁩ ⁨hours⁩ ago
Keep in mind you can’t do a migration with device passthrough, even if the VM is shut down.

I use zfs sync every 15 minutes for my replication. No issues.

original
LordCrom@lemmy.world ⁨9⁩ ⁨hours⁩ ago
For iSCSI, ita best to dedicate a port from each host and enable jumbo frames (mtu 9000) to get the most out of it.

Also try to dedicate a port for proxmox vm usage. Vnet off these 3 connections too.

original
- non_burglar@lemmy.world ⁨7⁩ ⁨hours⁩ ago
  
  jumbo frames (mtu 9000) to get the most out of it
  
  On dedicated network segments where only jumbo frames are configured at broadcast points, like a dedicated SAN, sure.
  
  Not the best for multi-mode networks like most home labs.
  
  iSCSI is already good at shoveling frames. The tiny benefit one can get from using jumbo frames isn’t really worth the problems it causes in multi mode networks.
  
  original
skankhunt42@lemmy.ca ⁨13⁩ ⁨hours⁩ ago
I don’t know about proxmox but I have 6 4 8GB, 5 lenovo Tiny (i5-4x and i7-8x), and 3 laptops in a k8s cluster and love it. I actually have too many hosts but it’s a lot of fun for me to play on them. Storage is longhorn for anything with a database, everything else is over NFS onto a box I built myself. If you use the *arrs try them over the network and watch the logs. They HATE network storage and will complain a lot. If you don’t have any issues there you should be good.

I haven’t had any issues and just use the 1gb network. As for form factor, I think it’s whatever you have room for. The laptops are annoying but they have the best hardware so I put up with it.

original
- buildmylab@lemmus.org ⁨13⁩ ⁨hours⁩ ago
  Appreciate the reply! I’ve been running this Tiny for 2 years now and it’s been rock solid. My main hesitation is really finding the right price-to-performance sweet spot between compute and storage. Does this 3-node + Synology setup feel like a sensible middle ground, or am I overpaying/overthinking it?
  
  original
  - skankhunt42@lemmy.ca ⁨12⁩ ⁨hours⁩ ago
    Everything is super expensive these days which is why I’ve got all kinds of nodes. I lurk marketplace, kijiji, etc for “broken” or “for parts” nodes. Laptops with broken keyboards, screens, no battery, etc are perfect for me. I7 11th gen was $90 CAD. Works perfectly for a node. Found a cheap tiny with a locked bios. Removed the CMOS battery to reset and I’m good to go.
    
    If you’re not willing to do that, the working tinys are perfect in my opinion.
    
    The working Synology seems extra to me. You could use software like ceph, or diy it. Storage boxes don’t need crazy specs, just sata ports and power. The only reason I’d consider it is if you just want something “simple that works” and you’re willing to pay for it.
    
    original
    -> View More Comments
ryan_@piefed.social ⁨10⁩ ⁨hours⁩ ago
I have a similar setup, 3 mini pcs (2 dell, 1 hp) with a second 2.5Gb interface on each and a synology nas.

Initially I used iscsi for storage but I had a couple of annoyances that I encountered:

At that time, taking snapshots on vms using iscsi storage wasn’t possible.

My Synology NAS doesn’t have a way to natively install a 2.5Gb interface, so I used a USB 2.5Gb NIC. Occasionally (like one or twice per year, from my recollection) the NIC would stop working and my VMs would lose their storage. Easily fixed by rebooting the Synology and VMs, but annoying none the less.

Because of that I ended up installing an SSD in each node and creating a CEPH storage cluster. No more issues with VMs losing storage and it gave me the ability to create snapshots.

My current environment is 17 LXCs and 4 VMs living in a ceph cluster that runs on each proxmox node. Using PBS, I have backup jobs scheduled across 4 days each week, 2 of the days backup to my nas and the other 2 backup to Backblaze.

I’ve never encountered any noticeable bottlenecks with my current environment so I think you’ll be just fine
original