Rsync, syncthing, backups, mp3s, photos, json files; idk, a lot of tasks involve large amounts of small files. I personally ran into this problem training models on millions of photos. My GPUs would only get up to 25% utilization with mirrored HDDs, so I had to switch to SSDs.
sobchak@programming.dev 3 days ago
Rsync, syncthing, backups, mp3s, photos, json files; idk, a lot of tasks involve large amounts of small files. I personally ran into this problem training models on millions of photos. My GPUs would only get up to 25% utilization with mirrored HDDs, so I had to switch to SSDs.
Gladaed@feddit.org 3 days ago
Why are you doing that on a network storage as opposed to on device?
sobchak@programming.dev 3 days ago
Not enough room in the GPU machine for all the HDDs I needed.
People working on biological datasets.
Gladaed@feddit.org 2 days ago
Why are you doing that recreationally? How are you different from a researcher?