Jesus, my pool takes a little over a day, but I’ve only got around 100 gb how big is your pool?
Comment on Say Hello to the World's Largest Hard Drive, a Massive 36TB Seagate
needanke@feddit.org 3 weeks ago
What is the usecase for drives that large?
I ‘only’ have 12Tb drives and yet my zfs-pool already needs ~two weeks to scrub it all. With something like this it would literally not be done before the next scheduled scrub.
tehn00bi@lemmy.world 3 weeks ago
needanke@feddit.org 3 weeks ago
The pool is 30 TB with 20 filled
dirtbiker509@lemmy.world 3 weeks ago
Something is very wrong if it’s taking 2 weeks to scrub that.
remon@ani.social 3 weeks ago
Sounds like something is off. I have 20TB drives (x8, raid 6) … scrubbing takes less then 3 days.
Appoxo@lemmy.dbzer0.com 3 weeks ago
High capacity storage pools for enterprises.
Space is at a premium. Saving space should/could equal to better pricing/availability.frezik@lemmy.blahaj.zone 3 weeks ago
Not necessarily.
The trouble with spinning platters this big is that if a drive fails, it will take a long time to rebuild the array after shoving a new one in there. Sysadmins will be nervous about another failure taking out the whole array until that process is complete, and that can take days. There was some debate a while back on if the industry even wanted spinning platters >20TB. Some are willing to give up density if it means less worry.
I guess Seagate decided to go ahead, anyway, but the industry may be reluctant to buy this.
Appoxo@lemmy.dbzer0.com 3 weeks ago
I would assume with arrays they will use a different way to calculate parity or have higher redundancy to compensate the risk.
frezik@lemmy.blahaj.zone 3 weeks ago
If there’s higher redundancy, then they are already giving up on density.
We’ve pretty much covered the likely ways to calculate parity.
SuperUserDO@sh.itjust.works 3 weeks ago
There is an enterprise storage shelf (aka a bunch of drives that hooks up to a server) made by Dell which is 1.2 PB (yes petabytes). So there is a use, but it’s not for consumers.
grue@lemmy.world 3 weeks ago
That’s a use-case for a fuckton of total capacity, but not necessarily a fuckton of per-drive capacity. I think what the grandparent comment is really trying to say is that the capacity has so vastly outstripped mechanical-disk data transfer speed that it’s hard to actually make use of it all.
For example, let’s say you have these running in a RAID 5 array, and one of the drives fails and you have to swap it out. At 190MB/s max sustained transfer rate (figure for a 28TB Seagate Exos; I assume this new one is similar), you’re talking about over two days just to copy over the parity information and get the array out of degraded mode! At some point these big drives stop being suitable for that use-case just because the vulnerability window is so large that the risk of a second drive failure causing data loss is too great.
needanke@feddit.org 3 weeks ago
Thats exactly what I wanted to say, yes :D.
SuperUserDO@sh.itjust.works 3 weeks ago
I get it. But the moment we invoke RAID, or ZFS, we are outside what standard consumers will ever interact with, and therefore into business use cases. Remember, even simple homelab use cases involving docker are well past what the bulk of the world understands.
Bael422@lemmy.world 3 weeks ago
It’s to play Ark: Survival Evolved.
Aermis@lemmy.world 3 weeks ago
What’s scrubbing for?
gagootron@feddit.org 3 weeks ago
Aermis@lemmy.world 3 weeks ago
I’m not in the know of having your own personal data centers so I have no idea. … But how often is this necessary? Does accessing your own data on your hard drive require a scrub? I just have a 2tb on my home pc. Is the equivalent of a scrub like a disk clean up?
gagootron@feddit.org 3 weeks ago
You usually scrub you pool about once a month, but there are no hard rules on that. Accessing the data does not need a scrub, it is only a routine maintenance task. A scrub is not like a disk cleanup. With a disk cleanup you remove unneeded files and caches, maybe de-fragment as well. A scrub on the other hand validates that the data you stored on the pool is still the same as before. This is primarily to protect from things like bit rot.
There are many ways a drive can degrade. Sectors can become unreadable, random bits can flip, a write can be interrupted by a power outage, etc. Normal file systems like NTFS or ext4 can only handle this in limited ways. Mostly by deleting the corrupted data.
ZFS on the other hand is built using redundant storage. Storing the data spread over multiple drives in a special way allowing it to recover most corruption and even survive the complete failure of a disk. This comes at the cost of losing some capacity however.
Jimmycakes@lemmy.world 3 weeks ago
Data centers???
Hugin@lemmy.world 3 weeks ago
I worked on a terrain render of the entire planet. We were filling three 2 Tb drives a day for a month. So this would have been handy.
Hadriscus@jlai.lu 3 weeks ago
It’s like the petronas towers, everytime they’re finished cleaning the windows they have to start again
ipkpjersi@lemmy.ml 3 weeks ago
What drives do you have exactly? I have 7x6TB WD Red Pro drives and I can do a scrub less than 24 hours.
needanke@feddit.org 3 weeks ago
I have 2*12TB whitelabel WD drives (harvested from external drives but Datacenter drives accourding to the SN) and one 16 TB Toshiba white-label (purchased directly also meant for datacenters) in a raidz1.
How full is your pool? I have about 2/3rds full which impacts scrubbing I think. I also frequently access the pool which delays scrubbing.
ipkpjersi@lemmy.ml 3 weeks ago
It’s like 90% full, scrubbing my pool is always super fast.
pyre@lemmy.world 3 weeks ago
there was a time i asked this question about 500 megabytes
MystikIncarnate@lemmy.ca 3 weeks ago
I too, am old.
pyre@lemmy.world 3 weeks ago
I’m older than that but didn’t want to self report. the first hard disk i remember my father buying was 40mb.
MystikIncarnate@lemmy.ca 3 weeks ago
I remember renting a game, and it was on a high density 5.25" inch floppy at a whopping 1.2MB; but or family computer only had a standard density 5.25".
So we went to the neighbors house, who was one of the first computer nerds (I’m not sure he’s still alive now), who copied the game to a 3.5" high density 1.44MB disk, then we returned the rental because we couldn’t play it on the 1.2 MB HD 5.25" floppy.
… And that was the first time I was party to piracy.
needanke@feddit.org 3 weeks ago
I am not questioning the need for more storage but the need dor more storage without increased speeds.