How to make setup more resilient? Proxmox mini-PC \w iSCSI to TrueNAS

Submitted ⁨⁨1⁩ ⁨year⁩ ago⁩ by ⁨unsaid0415@szmer.info⁩ to ⁨selfhosted@lemmy.world⁩

Hello.

My setup is: Lenovo M920q mini pc with Proxmox installed (this doesn’t have IPMI, only vPRO and it’s annoying me) Fujitsu TX1320 M3 with TrueNAS Core installed - ZFS + RAID1 (this is a low-end “enterprise grade” server, and best thing - it has IPMI).

The Proxmox PC keeps all its CTs and 1 VM on the TrueNAS using iSCSI.

The idea behind my setup was that it felt nice that the TrueNAS would handle all the storage heavy lifting - ZFS, RAID etc., while the Proxmox mini PC would be a “compute-only” node that has a naked Proxmox install with some config.

The problem with that is if the TrueNAS machine loses power or is restarted, the Proxmox CTs/VMs switch their filesystem to read-only and stop responding to requests. This is because the iSCSI connection is interrupted. When the TrueNAS is back online, Proxmox doesn’t make any attempt to restart the VMs/CTs - they’d still be broken.

It’s annoying to me to have to VPN to the Proxmox web ui and wait 15 minutes until all the CTs/VMs are restarted and now again functioning on the “alive” iSCSI connection.

I was wondering what are my options here to remove the dependency chain?

I’m really into the idea of decomissioning the Proxmox node because I’m scared I won’t be able to (over VPN) change the power state of the machine if something goes wrong, since it only has vPro and not iSCSI like the TrueNAS machine.

Options I can think of:

Decomission the Proxmox node and move all Debian VMs/CTs to TrueNAS BSD jails. Is that even possible? Will all my Debian VMs work in BSD?
Decomission the Proxmox node, switch TrueNAS Core to TrueNAS Scale and move CTs/VMs to TrueNAS Scale’s Linux VMs
Keep the Proxmox node and somehow figure out how to get Proxmox to refresh the CTs/VMs on iSCSI connection loss.
Keep the Proxmox PC, but switch it to iESXI hoping that it handles the iSCSI failure more gracefully

source

Comments

Sort:hotnew top

ssdfsdf3488sd@lemmy.world ⁨1⁩ ⁨year⁩ ago
Get rid of iscsi. Instead, use truenas scalr for nas and use an ivol to run a vm of proxmox backup server. Run proxmox on the other box with oocal vms and just backup the vms to peoxmox backup server at aratr you are comfortable with (i.e. once a night). Map nfs shares from truenas i to any docker things you are running on your vms. This is way more resilient, gets local nvme speeds for the vms and still keepa the bulk of your files on the nas.

source
- MigratingtoLemmy@lemmy.world ⁨1⁩ ⁨year⁩ ago
  For a home environment, this is the correct idea
  
  source
  - ssdfsdf3488sd@lemmy.world ⁨1⁩ ⁨year⁩ ago
    I would argue it’s the correct idea up to a fairly decently sized business. Basically anything where you don’t have the budget or the need for super fault tolerant systems (i.e. where it’s ok to very rarely have a 20 minute to an hour outage in order to save 50k+ of IT hardware costs). You can take the above and go next step to a high availability proxmox cluster to further reduce potential downtime before you step into the realm of needing vmware and very expensive highly available and fast storage as well. It gets even more true when you start messing around with truenas and differential speed vdevs (i.e build a super fast nvme one with 10-25gig networking for some applications, a cheaper spinning rust one with maybe 10 gig networking for bulk storage. It’s also nice that, by using proxmox backup server as a zvol you can take advantage of all the benefit of both zfs replication/snapshotting and cloud (jstor/wasabi s3 bucket, another truenas server at a different location) for that zvol as well as your other data you are sharing as datasets.
    
    source
    -> View More Comments
melmi@lemmy.blahaj.zone ⁨1⁩ ⁨year⁩ ago
What is TrueNAS adding to this arrangement? Generally when people run two different servers at home, they keep the VM drives on the hypervisor and just use the NAS for storing bigger things like media files. Hosting VM drives over iSCSI works in an enterprise environment, but if you can’t guarantee uptime for your storage solution then you’re just adding failure modes.

It seems to me that your best bet is to go down to one server, which means cutting out either TrueNAS or Proxmox. Both can handle both storage (ZFS included!) and VMs, so ultimately it’s a matter of which you like better.

source
- unsaid0415@szmer.info ⁨1⁩ ⁨year⁩ ago
  I edited my post to clarify that TrueNAS keeps more than just VMs. It has photos, documents etc. as well.
  
  Generally when people run two different servers at home, they keep the VM drives on the hypervisor and just use the NAS for storing bigger things like media files
  
  This is simple and makes sense as well. My TrueNAS is only 2 HDDs, which is not ideal for VMs. I could get a larger drive SSD/M.2 drive for the hypervisor, though the Lenovo M920q supports 1xM.2 and 1x2.5" drive.
  
  Hosting VM drives over iSCSI works in an enterprise environment, but if you can’t guarantee uptime for your storage solution then all you’re doing is adding failure modes.
  
  Well, my whole setup comes from the fact that I wanted to cosplay as an enterprise environment (famous last words for a homelabber). I’ve been powering the TrueNAS up and down a lot due to some electricity-related construction in my apartament, and it brought out this flaw in my setup. I guess an UPS would be in order, as another poster pointed out.
  
  source
  - SeeJayEmm@lemmy.procrastinati.org ⁨1⁩ ⁨year⁩ ago
    
    Well, my whole setup comes from the fact that I wanted to cosplay as an enterprise environment
    
    I feel that. Experimenting I get, but I’d never have it be my primary vm backing storage. Esp. not on a 1Gb network, no ups, no redundancy.
    
    Someone else suggested local vm storage and a PBS VM on the TrueNas box. I think that’s a solid solution to consider.
    
    source
SheeEttin@lemmy.world ⁨1⁩ ⁨year⁩ ago
2 is the only possible one of those options.

You could also make the current setup more reliable by adding a UPS and/or second storage node for redundancy, so that when one goes down, the other is still available. Presumably TrueNAS supports this.

But nothing is going to help you recover if the iSCSI link is broken. It’s up to the host and guest OS to re-establish the link, and to the guest it usually looks like the hard drive has been unplugged, and I don’t know any OS that considers that a supported and recoverable condition.

source
- unsaid0415@szmer.info ⁨1⁩ ⁨year⁩ ago
  Thanks for making it clear that iSCSI power down is in fact one of the more grim scenarios, I couldn’t make it out how bad of a situation it is. In an enterprise environment a SAN being down would require some type of incident report.
  
  UPS - as you suggested - would solve most of my problems to be honest.
  
  source
TechAdmin@lemmy.world ⁨1⁩ ⁨year⁩ ago
VMs in ESXi have the same behavior when iSCSI connection is lost then restored later. Windows with iSCSI drive mounts shows the same behavior in that scenario too.

UPS would be a great addition no matter what option you choose.

source
Decronym@lemmy.decronym.xyz ⁨1⁩ ⁨year⁩ ago
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

Fewer Letters More Letters

LXC Linux Containers

NAS Network-Attached Storage

SSD Solid State Drive mass storage

[Thread #122 for this sub, first seen 9th Sep 2023, 18:55] [FAQ] [Full list] [Contact] [Source code]

source
unsaid0415@szmer.info ⁨1⁩ ⁨year⁩ ago
It’d be nice to have Proxmox and TrueNAS side by side on one machine, but since TrueNAS forums are against the virtualization of TrueNAS I’m somewhat stuck with having to have one bare metal machine per appliance.

source
- Decipher0771@lemmy.ca ⁨1⁩ ⁨year⁩ ago
  I’m sure you’ve heard plenty through the forums, but Truenas virtualized is perfectly fine so long as you’re passing through an HBA directly. It doesn’t affect reliability any, but it doesn’t add any features either.
  
  “Can I virtualized Truenas” is probably the second most popular question after “do I really need ECC ram”
  
  source
- Kaavi@lemmy.world ⁨1⁩ ⁨year⁩ ago
  I took my truenas some months ago and installed proxmox on that server instead.
  
  I can’t from your post figure out what you need truenas for? Proxmox runs zfs, etc?
  
  I made an lxc container with samba to share the discs in proxmox.
  
  source
  - unsaid0415@szmer.info ⁨1⁩ ⁨year⁩ ago
    I edited my post to clarify. TrueNAS also keeps documents, photos, torrents, music. I also use the mount feature so that the music server LXC can access music
    
    source
unsaid0415@szmer.info ⁨1⁩ ⁨year⁩ ago
Hey, OP here again.

Here’s what I ended up with:

upgrading my TrueNAS CORE to TrueNAS SCALE - it was really easy, just upload a 1.3GB upload file through the web UI. CORE’s apps/plugins are based on BSD jails, where SCALE apps are based on Kubernetes/Docker, so I can any arbitrary Docker container from Dockerhub as I please, rather than being limited to BSD jails

migrating all the VMs/LXCs to matching TrueNAS SCALE Applications. So e.g. my hand-made Navidrome LXC was migrated to the TrueNAS SCALE Application. Sometimes there was no equivalent TrueNAS app for what I was using - e.g. Forgejo, so I just ran an arbitrary container from dockerhub.

decomissioning the Proxmox mini-pc (Lenovo M920q). I’ll sell it later or maybe turn it into a pfSense router.

I installed a custom TrueNAS app repository called Truecharts. It has some apps that the default repo doesn’t have, and it also has a nice integration with Ingress (Traefik), which allows you to easily create a reverse proxy using just the GUI.

I’m still yet to figure out how to set up Let’s Encrypt for the services I made available to the Internet. I can no longer do things the Linux way, i must do it the Kubernetes way, so I’m kind of limited. Looks like HTTP01 challenges don’t work yet and I’ll have to use DNS01.
source

Fewer Letters	More Letters
LXC	Linux Containers
NAS	Network-Attached Storage
SSD	Solid State Drive mass storage