Apparent issues with ZFS on RPi 5

Submitted ⁨⁨3⁩ ⁨weeks⁩ ago⁩ by ⁨ramenshaman@lemmy.world⁩ to ⁨selfhosted@lemmy.world⁩

I tried to find a more applicable community to post this to but didn’t find anything.

I recently set up a NAS/server on a Raspberry Pi 5 (see my last post) and since then I’ve got everything installed into a 3D printed enclosure and I’ve got RAID set up (ZFS RAIDz1). Prior to setting up RAID, I could transfer files to/from the NAS at around 200MB/s, but now that RAID is seemingly working things are transferring at around 28-30 MB/s. I did a couple searches and found someone suggesting to disable sync ($ sudo zfs set sync=disabled zfspool). I tried that and it doesn’t seem to have had any effect. Any suggestions are welcome but keep in mind that I barely know what I’m doing.

source

Comments

Sort:hotnew top

3dcadmin@lemmy.relayeasy.com ⁨3⁩ ⁨weeks⁩ ago
A Pi with a hat and 5 sata drives isn’t ever going to perform very well with zfs in raid z1. cpu limited, io limited possibly even ram limited quite quickly. I love zfs but it a bit of a performance hog, though pretty bulletproof if setup correctly! the hat will be through a pci x1 interface, so at least set the pci to gen3 speeds. Jeff tells you about it here

jeffgeerling.com/…/forcing-pci-express-gen-30-spe…

source
- ramenshaman@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
  People keep saying the Pi is CPU limited but it hasn’t gotten over 10% between all 4 cores while I’ve been transferring files. 16GB of RAM doesn’t seem very limited to me either. Thanks for the link!
  
  source
  - HiTekRedNek@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
    It’s bus limited, not necessarily CPU limited.
    
    source
  - 3dcadmin@lemmy.relayeasy.com ⁨2⁩ ⁨weeks⁩ ago
    It isn’t that limited, but for zfs arm seems to perform much worse. Plus you often don’t get a full idea of true system load. the biggest limitation is the io, it is very bad for 5 drives in zfs raidz1. the data is distributed across all 5 drives with parity as well. the pi can only do around 500 meg transfers for an nvme drive whilst many other platforms will see 3000 meg, that is why it suffers so much in this case as that 500 meg is across 5 drives. tops you’d get is 10 meg transfer I reckon and that is roughly what you are seeing. you’d be better off with 3 larger drives in raidz1
    
    source
calamityjanitor@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
sync=disabled will make ZFS write to disk every 5 seconds instead of when software demands it, which maybe explains your LED behavior.

Jeff Geerling found that writes with Z1 was 74 MB/sec using the Radxa Penta SATA HAT with SSDs. Any HDD should be that fast, the SATA hat is likely the bottleneck.

Are you performing writes locally, or over smb?

Can try iostat or zpool iostat to monitor drive writes and latencies, might give a clue.

How much RAM does the Pi 5 have?

source
- calamityjanitor@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
  After some googling:
  
  Some Linux distributions (at least Debian, Ubuntu) enable init_on_alloc option as security precaution by default. This option can help to prevent possible information leaks and make control-flow bugs that depend on uninitialized values more deterministic.
  
  Unfortunately, it can lower ARC throughput considerably (see bug).
  
  If you’re ready to cope with these security risks 6, you may disable it by setting init_on_alloc=0 in the GRUB kernel boot parameters.
  
  I think it’s set to 1 on Raspberry Pi OS, you set it in /boot/cmdline.txtI think.
  
  [Exhaustive ZFS performance tuning guide](openzfs.github.io/openzfs-docs/Performance and Tuning/Workload Tuning.html)
  
  source
- ramenshaman@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
  I haven’t seen more than maybe 32MB/s. The transfers I’ve done are all on my local network from my desktop to my NAS which is plugged into my router. I have samba installed, my NAS shows up as a network drive just fine and I’ve just been dragging/dropping on my desktop GUI. Before setting up RAID, when I did this I would get around 200MB/s. The Pi has 16GB of RAM and I’m using less than 1.5GB.
  
  source
  - Lemmchen@feddit.org ⁨3⁩ ⁨weeks⁩ ago
    Those 200MB/s probably weren’t synchronous transfers. The OS tells you the write was complete, but it actually hasn’t committed the data to disk yet. (Wild guess)
    
    source
  - calamityjanitor@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
    Ah kay, definitely not a RAM size problem then.
    
    iostat -x 5 Will print out per drive stats every 5 seconds. The first output is an average since boot. Check all of the drives have similar values while performing a write. Might be one drive is having problems and slows everything down, hopefully unlikely if they are brand new drives.
    
    zpool iostat -w Will print out a latency histogram. Check if any have a lot above 1s and if it’s in the disk or sync queues. Here’s mine with 4 HDDs in z1 working fairly happily for comparison: Here’s mine with 4 HDDs in z1 working fairly happily for comparison
    
    The init_on_alloc=0 kernel flag I mentioned below might still be worth trying.
    
    source
HiTekRedNek@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
When people talk about CPU limitations on the rPi, they aren’t talking about just the actual processing portion of the machine. There are also a lot of other corners cut for basically all SBCs. Including bus width and throughput.

The problem is that when you use a software raid, like ZFS, or it’s precursors, you are using far more than the CPU. You’re also using the data bus between the CPU and the IO controller.

“CPU usage” indicators don’t really tell you how active your data buses are, but how active your CPU is, in having to process information.

Basically, it’s the difference between IO wait states, and CPU usage.

The Pi is absolutely a poor choice for input/output, period. Regardless of your “metrics” tell you, it’s data bus simply does not have the bandwidth necessary to control several hard drives at once with any sort of real world usability.

You’ve wasted your money on an entire ecosystem by trying to make it do something it wasn’t designed, nor has the capability, to do.

source
Zanathos@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
What type of disk (HDD or SSD) and how many disks in the pool?

RAIDZ1 configuration will bring your write speed down some due to data having to write to multiple disks at a time. This is true for most any RAID. Once written, your read speeds should remain the same or improve a bit though.

source
- ramenshaman@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
  I’ve got 5x 8TB HDD at 5400rpm
  
  source
TCB13@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
It’s a Pi, what are you expecting. You just wasted a ton of money on inferior hardware with extra software issues. You could’ve just got a mini pc with 2 nvme slots instead for half the price and add a 6 port sata board for 20$ to one of those. Much cheaper, way more reliable, upgradable and ZFS actually would’ve work as you expect.

source
- ramenshaman@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
  Just getting my feet wet. I have plenty of other uses for a Pi so if the NAS doesn’t work out that’s fine, I’ll get something else. Worst case I’m out $70-ish. Something is clearly not right with my config, I haven’t gotten the Pi to break a sweat yet. Thanks for your input though.
  
  source
Creat@discuss.tchncs.de ⁨3⁩ ⁨weeks⁩ ago
What size is the ARC set to? I’ve seen cases where it was fully disabled, which seemed to murder performance and is probably even worse when in such a CPU limited platform.

source
- ramenshaman@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
  I don’t know what ARC is and my searches so far haven’t helped.
  
  source
  - 3dcadmin@lemmy.relayeasy.com ⁨3⁩ ⁨weeks⁩ ago
    this is the limits of a slow interface and 5 drives. See my other reply to enable faster pci speeds. because of how zfs works 5 drives is slower than 3, takes more cache and write speeds especially will be slower, quite a lot slower. with 5 drives and 16gb you can easily have a zfs cache of 12 gigs to help it along, i guess this is why you are getting large gaps between writes. as someone else said a pi doesn’t do well in this case but I reckon you can improve it. however as also said it is never going to be a speedy solution. secure and safe for data but not fast
    
    source
  - Lemmchen@feddit.org ⁨3⁩ ⁨weeks⁩ ago
    Do you have the 8GB version of the Pi 5? You shouldn’t set the ARC to 8GB then. Usually it about half of the available system RAM. I’d probably set it lower if there’s only 8GB available in total.
    
    source
    -> View More Comments
  - Creat@discuss.tchncs.de ⁨3⁩ ⁨weeks⁩ ago
    ARC is the in-memory cache used by ZFS. If it’s completely off the effect can be dramatic. Under no circumstances should a larger cache cause anything to get slower, ever. Even the raspi didn’t have memory that is that slow that this is a reasonable outcome.
    
    As a concrete example: I was recently working on a server where a maintenance task that should take like 12hrs or so at the worst somehow took 2 weeks (!) and still wasn’t finished. That was ARC being disabled.
    
    source
SheeEttin@lemmy.zip ⁨3⁩ ⁨weeks⁩ ago
What are you actually using for storage, and how is it connected?

source
- ramenshaman@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
  5x 8TB HDD at 5400rpm all hooked up to a Radxa Penta SATA hat.
  
  source
monkeyman512@lemmy.world ⁨2⁩ ⁨weeks⁩ ago
Other people have already talked about why you are having performance issues with the Pi. As for a better NAS solution you will probably be better off with a used desktop PC from the last 10 years. If the computer doesn’t have enough SATA ports you can get a sata addon card or HBA (host buss adapter) addon card flashed in IT mode. You should be able to find a lot of options on eBay. Maybe people can chime in with specific models to look at.

source
kata1yst@sh.itjust.works ⁨3⁩ ⁨weeks⁩ ago
What controller?

source
- Faceman2K23@discuss.tchncs.de ⁨2⁩ ⁨weeks⁩ ago
  the radxa penta is a JMB585 connected with pcie gen3x1.
  
  They do overheat though, might need a heatsink and some ZFS tuning.
  
  source
- 3dcadmin@lemmy.relayeasy.com ⁨2⁩ ⁨weeks⁩ ago
  The controller is the built in Pi controller in the SoC - the hat just sorts out the logic to route the 5 sata ports
  
  source
- ramenshaman@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
  I guess the Radxa Penta SATA hat would be considered the controller.
  
  source