lightrush
@lightrush@lemmy.ca
- Comment on Mirror seeing half the write IOPS on one disk than the other, is this normal? 2 months ago:
I think the board has reached the end of the road. 😅
- Comment on Mirror seeing half the write IOPS on one disk than the other, is this normal? 2 months ago:
Iiinteresting. I’m on the larger AB350-Gaming 3 and it’s got REV: 1.0 printed on it. No problems with the 5950X so far. 🤐
- Comment on Mirror seeing half the write IOPS on one disk than the other, is this normal? 2 months ago:
On paper it should support it. I’m assuming it’s the ASRock AB350M. With a certain BIOS version of course.
- Comment on Mirror seeing half the write IOPS on one disk than the other, is this normal? 2 months ago:
B350 isn’t a very fast chipset to begin with
For sure.
I’m willing to bet the CPU in such a motherboard isn’t exactly current-gen either.
It’s a Ryzen 9 5950X and I’m pretty proud how far I’ve managed to stretch this board. 😆
Are you sure you’re even running at PCIe 3.0 speeds too?
So given the CPU, it should be PCIe 3.0, but that doesn’t remove any of the queues/scheduling suspicions for the chipset.
- Comment on Mirror seeing half the write IOPS on one disk than the other, is this normal? 2 months ago:
I put the low IOPS disk in a good USB 3 enclosure, hooked to an on-CPU USB controller. Now things are flipped:
capacity operations bandwidth pool alloc free read write read write ------------------------------------ ----- ----- ----- ----- ----- ----- storage-volume-backup 12.6T 3.74T 0 563 0 293M mirror-0 12.6T 3.74T 0 563 0 293M wwn-0x5000c500e8736faf - - 0 406 0 146M wwn-0x5000c500e8737337 - - 0 156 0 146M
You might be right about the link problem.
Looking at the B350 diagram, the whole chipset is hooked via PCIe 3.0 x4 link to the CPU. The other pool (the source) is hooked via USB controller on the chipset. The SATA controller is also on the chipset so it also shares the chipset-CPU link. I’m pretty sure I’m also using all the PCIe links the chipset provides for SSDs. So that’s 4GB/s total for the whole chipset. Now I’m probably not saturating the whole link, in this particular workload, but perhaps there’s might be another related bottleneck.
- Comment on Mirror seeing half the write IOPS on one disk than the other, is this normal? 2 months ago:
Turns out the on-CPU SATA controller isn’t available when the NVMe slot is used. 🫢 Swapped SATA ports, no diff. Put the low IOPS disk in a good USB 3 enclosure, hooked to an on-CPU USB controller. Now things are flipped:
capacity operations bandwidth pool alloc free read write read write ------------------------------------ ----- ----- ----- ----- ----- ----- storage-volume-backup 12.6T 3.74T 0 563 0 293M mirror-0 12.6T 3.74T 0 563 0 293M wwn-0x5000c500e8736faf - - 0 406 0 146M wwn-0x5000c500e8737337 - - 0 156 0 146M
- Comment on Mirror seeing half the write IOPS on one disk than the other, is this normal? 2 months ago:
Interesting. SMART looks pristine on both drives. Brand new drives - Exos X22. Doesn’t mean there isn’t an impending problem of course.
- Mirror seeing half the write IOPS on one disk than the other, is this normal?discourse.practicalzfs.com ↗Submitted 2 months ago to selfhosted@lemmy.world | 14 comments
- Comment on Recommendations for Hardware for Physical Media/Jellyfin Server 6 months ago:
- Lenovo ThinkCentre / Dell OptiPlex USFF machine like the M710q.
- Secondary SATA SSD for a RAID1 mirror
- External USB disks for storage
- WD Elements generally work well when well ventilated
- OWC Mercury Elite Pro Quad has a very well implemented USB path and has been problem-free in my testing
- Debian / Ubuntu LTS
- ZFS for the disk storage
Here’s an example:
- Comment on Slow sequential single file reads on ZFS 6 months ago:
OK, I think it may have to do with the odd number of data drives. If I create a raidz2 with 4 of the 5 disks, even with
ashift=12,recordsize=128K, the performance in sequential single thread read is stellar. What’s not clear is why this doesn’t affect, or not as much, the 4x 8TB-drive raidz1. - Submitted 6 months ago to selfhosted@lemmy.world | 2 comments
- Comment on Just a bunch of enclosures 7 months ago:
I’ve been on the USB train since 2019. You’re exactly right, you gotta get devices with good USB-to-SATA chipsets, and you gotta keep them cool. From all my testing I’ve discovered that:
- ASM1351 and ASM235CM are generally problem-free, but the former needs passive cooling if close to a disk. A small heatsink adhered with standard double-sided heat conductive tape is good enough.
- Host controllers matter too. Intel is generally problem-free. So is VIA. AMD has some issues on the CPU side on some models which are still not fully solved.
I like this box in particular because it uses a very straightforward design. It’s got 4x ASM235CM with cooling connected to a VIA hub. It’s got a built-in power supply, fan, it even comes with good cables. It fixes a lot of the system variables to known good values. You’re left with connecting it to a good USB host controller.
- Comment on Just a bunch of enclosures 7 months ago:
I thought about it, but it typically requires extra PCIe cards that I can’t rely on as there’s no space in one of the machines and no PCIe slots in the other. That’s why I did a careful search till I stumbled upon this particular enclosure and then I tested one with ZFS for over a week before buying the rest.
- Comment on Just a bunch of enclosures 7 months ago:
ASMedia ASM1351 (heatsinked) or ASM235CM on the device side 🥹
- Comment on Just a bunch of enclosures 7 months ago:
Thanks for the warning ⚠️🙏
This isn’t my first rodeo with ZFS on USB. I’ve been running USB for a few years now. Recently I ran this particular box through a battery of tests and I’m reasonably confident that with my particular set of hardware it’ll be fine. It passed everything I threw at it, once connected to a good port on my machine. But you’re generally right and as you can see I discussed that in the testing thread, and I encountered some issues that I managed to solve. If you think I’ve missed something specific - let me know! 😊
- Comment on Just a bunch of enclosures 7 months ago:
That was the cheapest option. 🤭
- Comment on Just a bunch of enclosures 7 months ago:
Two machines. A main server/workstation and a small off-site backup machine that runs the same services but hass less compute and RAM.
- Comment on Just a bunch of enclosures 7 months ago:
- 8x 8TB in a set of 2
- 5x 16TB in a set of 2
- Submitted 7 months ago to selfhosted@lemmy.world | 26 comments
- Comment on me too 7 months ago:
It can’t because birds are not real.
- Submitted 7 months ago to selfhosted@lemmy.world | 0 comments
- Comment on Bowl cut gang, rise up. 11 months ago:
It never was “bowl cut.”
