Proxmox Disk Performance Problems

Submitted ⁨⁨6⁩ ⁨months⁩ ago⁩ by ⁨SeeJayEmm@lemmy.procrastinati.org⁩ to ⁨selfhosted@lemmy.world⁩

I’ve started encountering a problem that I should use some assistance troubleshooting. I’ve got a Proxmox system that hosts, primarily, my Opnsense router. I’ve had this specific setup for about a year.

Recently, I’ve been experiencing sluggishness and noticed that the IO wait is through the roof. Rebooting the Opnsense VM, which normally only takes a few minutes is now taking upwards of 15-20. The entire time my IO wait sits between 50-80%.

The system has 1 disk in it that is formatted ZFS. I’ve checked dmesg, and the syslog for indications of disk errors (this feels like a failing disk) and found none. I also checked the smart statistics and they all “PASSED”.

Any pointers would be appreciated.

Example of my most recent host reboot.

source

Comments

Sort:hotnew top

tvcvt@lemmy.ml ⁨6⁩ ⁨months⁩ ago
There was a recent conversation on the Practical ZFS discourse site about poor disk performance in Proxmox (discourse.practicalzfs.com/t/…/1421/). Not sure if you’re seeing the same thing, but it could be that your VMs are running into the same too-small volblocksize that PVE uses to make zvols for its Vans under ZFS.

If that’s the case, the solution is pretty easy. In your PVE datacenter view, go to storage and create a new ZFS storage pool. Point it to the same zpool/dataset as the one you’ve already got and set the block size to something like 32k or 64k. Once you’ve done that, move the VM’s disk to that new storage pool.

Like I said, not sure if you’re seeing the same issue, but it’s a simple thing to try.

source
- SeeJayEmm@lemmy.procrastinati.org ⁨6⁩ ⁨months⁩ ago
  This was really interesting, thanks for the info.
  
  source
scrubbles@poptalk.scrubbles.tech ⁨6⁩ ⁨months⁩ ago
(If it’s not failing, which would be the first thing I’d check)

Do you have any new VMs up and running. IO was the bane of my existence with proxmox, but realized it’s just that VMs eat a ton of IO, especially with ZFS. A standard HDD won’t cut it (unless you have one and only one VM using that disk). Even sata SSDs just didn’t cut it over time, I had to build a full raid that would support 5-10 VMs on it before I saw IO wait drop enough.

source
- SeeJayEmm@lemmy.procrastinati.org ⁨6⁩ ⁨months⁩ ago
  I’m trying to think of anything I may have changed since the last time I rebooted the opnsense VM. But I try to keep up on updates and end up rebooting pretty regularly. The only things on this system are the opnsense VM and a small pihole VM. At the time of the screenshot above, the opnsense VM was the only thing running.
  
  If it’s not a failing HDD, my next step is to try and dig into what’s generating the I/O to see if there’s something misbehaving.
  
  source
  - scrubbles@poptalk.scrubbles.tech ⁨6⁩ ⁨months⁩ ago
    I had bad luck with ZFS on proxmox because of all of the overhead, I found with my tiny cluster it was better to do good old ext4 and then just do regular backups. ZFS actually killed quite a few of my drives because of it’s heavyweight. Not saying that’s your problem, but I wouldn’t be surprised if it was
    
    source
seaQueue@lemmy.world ⁨6⁩ ⁨months⁩ ago
iowait is indicative of storage not being able to keep up with the performance of the rest of the system. What hardware are you using for storage here?

source
- SeeJayEmm@lemmy.procrastinati.org ⁨6⁩ ⁨months⁩ ago
  It’s an old Optiplex SFF with a single HDD. Again, my concern isn’t that it’s “slow”. It’s that performance has rather suddenly tanked and the only changes I’ve made are regular OS updates.
  
  source
  - seaQueue@lemmy.world ⁨6⁩ ⁨months⁩ ago
    If I had to guess there was a code change in the PVE kernel or in their integrated ZFS module that led to a performance regression for your use case. I don’t really have any feedback there, PVE ships a modified version of an older kernel (6.2?) so something could have been backported into that tree that led to the regression. Same deal with ZFS, whichever version the PVE folks are shipping could have introduced a regression as well.
    
    Your best bet is to raise an issue with the PVE folks after identifying which kernel version introduced the regression, you’ll want to do a binary search between now and the last known good time that this wasn’t occurring to determine exactly when the issue started - then you can open an issue describing the regression.
    
    Or just throw a cheap SSD at the problem and move on, that’s what I’d do here. Something like this should outlast the machine you put it in.
    
    source
    -> View More Comments
  - HybridSarcasm@lemmy.world ⁨6⁩ ⁨months⁩ ago
    Quick and easy fix attempt would be to replace the HDD with an SSD. As others have said, the drive may just be failing. Replacing with an SSD would not only get rid of the suspect hardware, but would be an upgrade to boot. You can clone the drive, or just start fresh with the backups you have.
    
    source
    -> View More Comments
pyrosis@lemmy.world ⁨6⁩ ⁨months⁩ ago
Out of curiosity what filesystem did you choose for you opnsense vm. Also can you tell if it’s a zvol, qcow2, or raw disk. FYI if it’s a qcow2 or a raw they both would benefit from a record size of 64k if they exist in a vm dataset. If it’s a zvol still 64 k can help.

I also utilize a heavily optimized setup running opnsense within proxmox. My vm filesystem is ufs because it’s on top of proxmox zfs. You can always find some settings in your opnsense vm to migrate log files to tmpfs which places them in memory. That will heavily reduce disk writes from opnsense.

source
- SeeJayEmm@lemmy.procrastinati.org ⁨6⁩ ⁨months⁩ ago
  Proxmox is using ZFS. Opnsense is using UFS. Regarding the record size I assume you’re referring to the same thing this comment is?
  
  You can always find some settings in your opnsense vm to migrate log files to tmpfs which places them in memory.
  
  I’ll look into this.
  
  source
  - pyrosis@lemmy.world ⁨6⁩ ⁨months⁩ ago
    I’m specifically referencing this little bit of info for optimizing zfs for various situations.
    
    Vms for example should exist in their own dataset with a tuned record size of 64k
    
    Media should exist in its own with a tuned record size of 1mb
    
    lz4 is quick and should always be enabled. It will also work efficiently with larger record sizes.
    
    Anyway all the little things add up with zfs. When you have an underlying zfs you can get away with more simple and performant filesystems on zvols or qcow2. XFS, UFS, EXT4 all work well with 64k record sizes from the underlying zfs dataset/zvol.
    
    Btw it doesn’t change immediately on existing data if you just change the option on a dataset. You have to move the data out then back in for it to have the new record size.
    
    source
    -> View More Comments
TheHolm@aussie.zone ⁨6⁩ ⁨months⁩ ago
Run long smart test on the disk and check smart data after that. Other possibility is ZFS pool is nearly full.

source

HybridSarcasm@lemmy.world ⁨6⁩ ⁨months⁩ ago

Check the ZFS pool status. You could lots of errors that ZFS is correcting.

source

SeeJayEmm@lemmy.procrastinati.org ⁨6⁩ ⁨months⁩ ago

I’m starting to lean towards this being an I/O issue but I haven’t figure out what or why yet. I don’t often make changes to this environment since it’s running my Opnsens router.

root@proxmox-02:~# zpool status
  pool: rpool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0B in 00:56:10 with 0 errors on Sun Apr 28 17:24:59 2024
config:

        NAME                                    STATE     READ WRITE CKSUM
        rpool                                   ONLINE       0     0     0
          ata-ST500LM021-1KJ152_W62HRJ1A-part3  ONLINE       0     0     0

errors: No known data errors

source

pyrosis@lemmy.world ⁨6⁩ ⁨months⁩ ago
It looks like you could also do a zpool upgrade. This will just upgrade your legacy pools to the newer zfs version. That command is fairly simple to run from terminal if you are already examining the pool.

source
- -> View More Comments

Max_P@lemmy.max-p.me ⁨6⁩ ⁨months⁩ ago
It could be a disk slowly failing but not throwing errors yet. Some drives really do their best to hide that they’re failing. So even a passing SMART test I would take with some salt.

I would start by making sure you have good recent backups ASAP.

You can test the drive performance by shutting down all VMs and using tools like fio to do some disk benchmarking. It could be a VM causing it. If it’s an HDD in particular, the random reads and writes from VMs can really cause seek latency to shoot way up. Could be as simple as a service logging some warnings due to junk incoming traffic, or an update that added some more info logs, etc.

source
- SeeJayEmm@lemmy.procrastinati.org ⁨6⁩ ⁨months⁩ ago
  
  I would start by making sure you have good recent backups ASAP.
  
  I do.
  
  Could be as simple as a service logging some warnings due to junk incoming traffic, or an update that added some more info logs, etc.
  
  Possible. It’s a really consistent (and stark) degradation in performance tho and is repeatable even when the opnsense VM is the only one running.
  
  source
possiblylinux127@lemmy.zip ⁨6⁩ ⁨months⁩ ago
Sounds like a hardware issue. I would also check dmesg and journalctl

source
catloaf@lemm.ee ⁨6⁩ ⁨months⁩ ago
Did you do a smart test?

source
- SeeJayEmm@lemmy.procrastinati.org ⁨6⁩ ⁨months⁩ ago
  Short test completed without error.
  
  source
- SeeJayEmm@lemmy.procrastinati.org ⁨6⁩ ⁨months⁩ ago
  Kinda feel dumb that my answer is no. Let me do that and report back.
  
  source
  - catloaf@lemm.ee ⁨6⁩ ⁨months⁩ ago
    While you’re waiting for that, I’d also look at the smart data and write the output to a file, then check it again later to see if any of the numbers have changed, especially reallocated sectors, pending sectors, corrected and uncorrected errors, stuff like that.
    
    Actually, I’m pretty sure that Proxmox will notify you of certain smart issues if you have emails configured.
    
    You should also shut the host down and reseat the drives, and check the cables to make sure they’re all properly seated too. It’s possible that one has come loose but not enough to drop the link.
    
    source
    -> View More Comments