Data HDD with SSD catch drive

Submitted ⁨⁨3⁩ ⁨weeks⁩ ago⁩ by ⁨rambos@lemm.ee⁩ to ⁨selfhosted@lemmy.world⁩

After upgrading my internet connection I immediatelly noticed that my HDD tops 40 MB/s and bottlnecking download speed in qbittorrent. Is it possible to use SSD drive as a catch drive for 12 TB HDD so it uses SSD speeds when downloading and moves files to HDD later on? If yes, does it make sense? Anyone using anything simmilar? Would 512 GB be enough or could I benefit from 2TB SSD?

HDD is just for jellyfin (movies/shows), not in raid, dont need backup for that drive, I can afford risking data if that matters at all

All suggestions are welcome, Thx in advance

source

Comments

Sort:hotnew top

ShortN0te@lemmy.ml ⁨3⁩ ⁨weeks⁩ ago
40MB/s is very very low even for a HDD. I would eventually debug why it’s that low.

Yes it’s possible. FS like zfs btrfs etc. support that.

source
- catloaf@lemm.ee ⁨3⁩ ⁨weeks⁩ ago
  It’s probably a 5400rpm drive, and/or SMR. Both are going to make it slower.
  
  source
  - Appoxo@lemmy.dbzer0.com ⁨3⁩ ⁨weeks⁩ ago
    5.4k + smr would explain it at write but not at read.
    
    source
  - Markaos@lemmy.one ⁨3⁩ ⁨weeks⁩ ago
    In my very limited experience with my 5400rpm SMR WD disk, it’s perfectly capable of writing at over 100 MB/s until its cache runs out, then it pretty much dies until it has time to properly write the data, rinse and repeat.
    
    40 MB/s sustained is weird (but maybe it’s just a different firmware? I think my disk was able to actually sustain 60 MB/s for a few hours when I limited the write speed, 40 could be a conservative setting that doesn’t even slowly fill the cache)
    
    source
- acosmichippo@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
  agreed, I think there is something else going on here. test the write speed with another application, I doubt the drive actually maxes out at 40MB/s unless it’s severely fragmented or failing.
  
  source
- rambos@lemm.ee ⁨3⁩ ⁨weeks⁩ ago
  Its the cheapest drive I could find (refurbished seagate from amazon), I thought thats the reason for being slow, but wasnt aware its that low. Im also getting 25-40 MB/s (200-320 Mbps) when copying files from this drive over network. Streaming works great so its not too slow at all. Is there better way of debugging this? What speeds can I expect from good drive or best drive?
  
  Ill research more about BTRFS and ZFS, thx
  
  source
  - acosmichippo@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
    can you copy files to it from another local disk?
    
    source
    -> View More Comments
- possiblylinux127@lemmy.zip ⁨3⁩ ⁨weeks⁩ ago
  Btrfs doesn’t support using a cache drive
  
  source
johntash@eviltoast.org ⁨3⁩ ⁨weeks⁩ ago
Unraid has this with their cache pools. ZFS can also be configured to have a cache drive for writes.

You can also DIY with something like mergerfs and separate file systems.

source
- rambos@lemm.ee ⁨3⁩ ⁨weeks⁩ ago
  Ive heard about all of these before, gonna do more research. Thank you
  
  source
braindefragger@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
Yes. It’s part of the application and well documented. What did you try and not work?

source
- rambos@lemm.ee ⁨3⁩ ⁨weeks⁩ ago
  Are you also talking about incomplete directory in qbit? Doesnt make it faster afaik, but I might be wrong. I havent tried anything yet, wanted to check is it something usual or not worth at all. Got zero experience with using SSD as catch drive, it just made sense to me
  
  source
  - braindefragger@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
    Yes, if the temporary directory where the files are being downloaded (incomplete folder) is on the SSD, then it should be faster, especially if you’ve identified a cheap HDD as your bottleneck. Unless you are incorrect about the HDD being the bottleneck.
    
    source
    -> View More Comments
slazer2au@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
You can and Qbittorrent has this functionality built in. You set your in progress download folder to be the SSD then set the move when completed to your HDD.

As for the size, that would depend on how much you are downloading.

source
- rambos@lemm.ee ⁨3⁩ ⁨weeks⁩ ago
  But that would first download to SSD, then move to HDD and then become available (arr import) on jellyfin server, making it slower than not using SSD. Am I missing something?
  
  source
  - BombOmOm@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
    The biggest thing is you have changed a random write to a linear write, something HDDs are significantly better at. The torrent is downloading little pieces from all over the place, requiring the HDD to move it’s head all over the place to write them. But when simply copying off the ssd, it keeps the head in roughly one place and just writes lineally, utilizing it’s maximum write speed.
    
    I would say try it out, see if it helps.
    
    source
    -> View More Comments
Lost_My_Mind@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
Great that you have a catch drive. I assume the data drive manages everything. So I’m going to call that the manager drive.

Now you just need:

A 1st base drive. A 2nd base drive. A 3rd base drive. A shortstop drive. A left Field drive. A center field drive. A right field drive. About 3-4 starting drives A half dozen reliever drives. A closer drive. A hittch coach drive And a couple of base running coach drives!

Got yourself a baseball team!

source
possiblylinux127@lemmy.zip ⁨3⁩ ⁨weeks⁩ ago
bcachefs will fill this role someday.

For now there is ZFS which as a cache drive option. Keep in mind it will absolutely destroy the cache drive by wearing out the flash

source
- rambos@lemm.ee ⁨3⁩ ⁨weeks⁩ ago
  Ill look into ZFS, but in meantime I found out my HDD is probably not bottleneck. Still want to learn about this so thanks for your comment
  
  source
- TheHolm@aussie.zone ⁨3⁩ ⁨weeks⁩ ago
  ZFS ZIL will not help in this case.
  
  source
Maxy@lemmy.blahaj.zone ⁨3⁩ ⁨weeks⁩ ago
qBittorrent has exactly the option you’re looking for, I believe it’s called “incomplete download path” in the settings, letting you store incomplete downloads at a temporary path and moving them to their regular location when the download finishes. Aside from the download speed improvement, this will also lead to less fragmentation on your HDD (which might be part of the reason why it is so slow when downloading directly to it). Pre-allocating space could have the same effect, but I would recommend only using one of these two solutions at once (pre-allocating space on your SSD would only waste space)

source
- rambos@lemm.ee ⁨3⁩ ⁨weeks⁩ ago
  But that would first download to SSD, then move to HDD and then become available (arr import) on jellyfin server, making it slower than not using SSD. Am I missing something?
  
  source
  - braindefragger@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
    
    Is it possible to use SSD drive as a catch drive for 12 TB HDD so it uses SSD speeds when downloading and moves files to HDD later on?
    
    Is that not what you asked for?
    
    source
    -> View More Comments
  - Maxy@lemmy.blahaj.zone ⁨3⁩ ⁨weeks⁩ ago
    It depends what you’re optimising for. If you want a single (relatively small) download to be available on your HDD as fast as possible, then your current setup might be better (optimising for lower latency). However, if you want to be maxing out your internet speeds at all time and increase your HDD speeds by making the copy sequential (optimising for throughput), then the setup with the catch drive will be better. Keep in mind that a HDD’s sequential write performance is significantly higher than its random write performance, so copying a large file in one go will be faster than copying a whole bunch of random chunks in a random order (like torrents do). You can check the difference for yourself by doing a disk benchmark and comparing the sequential vs random writes of your drive.
    
    source
    -> View More Comments
nitrolife@rekabu.ru ⁨3⁩ ⁨weeks⁩ ago
I used lvm with SSD cache few years, but time to time I have problems with loads after reboot. If forgot about reboots all work great with LVM raid + LVM cache. Cache can be configured without raid.

source
- rambos@lemm.ee ⁨3⁩ ⁨weeks⁩ ago
  
  cache few years, but time to time I have problems w
  
  Thx, Ill check it out
  
  source
capital@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
I do this with mergerfs.

I then periodically use their prewritten scripts to move things off the cache and to the backing drives.

source
- rambos@lemm.ee ⁨3⁩ ⁨weeks⁩ ago
  Thanks, Ill check mergefs
  
  source
fiddlesticks@lemmy.dbzer0.com ⁨3⁩ ⁨weeks⁩ ago
Depends on the file system, I know for a fact that ZFS supports ssd caches (in the form of l2arc and slog) and I believe that lvm does something similar (although I’ve never used it).

As for the size, it really depends how big the downloads are if you’re not downloading the biggest 4k movies in existence then you should be fine with something reasonably small like a 250 or 500gb ssd (although I’d always recommend higher because of durability and speed)

source
- rambos@lemm.ee ⁨3⁩ ⁨weeks⁩ ago
  Thx. I use ext4 right now. I might consider reformating, but so many new words to reasearch before deciding that. I heard about ZFS, but not sure is that right for me since I only have 16 GB of RAM.
  
  Downloads are 100-200 GB max, but less than 40 GB most of the time. I have 512 GB in use and 2TB SSD not in use, can swap them if needed
  
  source
- lemmylommy@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
  l2arc is a read cache. Slog only is for synchronous writes.
  
  source
Decronym@lemmy.decronym.xyz ⁨3⁩ ⁨weeks⁩ ago
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:

Fewer Letters More Letters

LVM (Linux) Logical Volume Manager for filesystem mapping

SSD Solid State Drive mass storage

ZFS Solaris/Linux filesystem focusing on data integrity

[Thread #938 for this sub, first seen 27th Aug 2024, 13:05] [FAQ] [Full list] [Contact] [Source code]

source
lemmylommy@lemmy.world ⁨3⁩ ⁨weeks⁩ ago
Any HDD should be able to get at least 100MB/s sequential write speed. Unfortunately torrent writes are usually very random, which just kills hdd performance. Multiple parallel downloads or concurrent playback from the same disk will only make it worse.

Using a SSD for temporary files will absolutely help. It should be big enough to hold all the files you are downloading at any one time.

You could also try to find a write cache setting that works for you. That way what would usually be many small writes can be combined to bigger chunks in memory before sending them to storage. Depending on how much ram is available I would start at 1GB or so and if it is still bottlenecking try in- or decreasing until it improves. Of course always stay in the range of free ram.

Back when I was torrenting (ages ago) write cache helped a lot. It should be somewhere in the settings menu.

source
- rambos@lemm.ee ⁨3⁩ ⁨weeks⁩ ago
  Oh, you are talking about torrent client settings? I could spare 1-2 GB of RAM, but not more than that (got 16 GB in total). I see this might help a lot, but I would I still be limited with HDD max write speed? Using SSD for temporary files sounds great, but waiting files to be coppied to HDD would slow it down if I understood correctly
  
  source

Fewer Letters	More Letters
LVM	(Linux) Logical Volume Manager for filesystem mapping
SSD	Solid State Drive mass storage
ZFS	Solaris/Linux filesystem focusing on data integrity