Slow sequential single file reads on ZFS

Submitted ⁨⁨6⁩ ⁨months⁩ ago⁩ by ⁨lightrush@lemmy.ca⁩ to ⁨selfhosted@lemmy.world⁩

I built a 5x 16TB RAIDz2, filled it with data, then I discovered something odd about it.

Sequentially reading a single file from the file system gave me around 40MB/s. Reading multiple in parallel brought the total throughput in the hundreds of megabytes - where I’d expect it. This is really weird. The 5 disks show 100% utilization during single file reads.

Splitting the the RAIDz2 into two RAIDz1s, or into one RAIDz1 and a mirror improved reads to 100 and something MB/s.

I have an existing RAIDz1 made of 4x 8TB disks on the same machine. That one reads with 250-350MB/s. I made an equivalent 4x 16TB RAIDz1 from the new drives and that read with about 100MB/s. Much slower.

All of this was done with ashift=12 and default recordsize.

I decided to try RAIDz2 with ashift=13 even though the disks really say they’ve got 4K physical block size. Lo and behold, the single file reads went to over 150MB/s. 🤔

Following from there, I got full throughput when I increased the recordsize to 1M. This produces full throughput even with ashift=12.

Can anyone make any sense of this?

source

Comments

Sort:hotnew top

NeoNachtwaechter@lemmy.world ⁨6⁩ ⁨months⁩ ago
Would you use zfs and raid-z when there is only 1 file on your disk?

Would you build 4 ticket counters when your concert hall has only 1 seat? Would you build a 4 lane highway when there is only 1 car in your country?

:-)

source
lightrush@lemmy.ca ⁨6⁩ ⁨months⁩ ago
OK, I think it may have to do with the odd number of data drives. If I create a raidz2 with 4 of the 5 disks, even with ashift=12, recordsize=128K, the performance in sequential single thread read is stellar. What’s not clear is why this doesn’t affect, or not as much, the 4x 8TB-drive raidz1.

source