Am I corrupting my data?

Submitted ⁨⁨2⁩ ⁨days⁩ ago⁩ by ⁨thelemonalex@lemmy.world⁩ to ⁨selfhosted@lemmy.world⁩

Hi everyone, I’ve been working on my homelab for a year and a half now, and I’ve tested several approaches to managing NAS and selfhosted applications. My current setup is an old desktop computer that boots into Proxmox, which has two VMs:

TrueNAS Scale: manages storage, shares and replication.
Debian 12 w/ docker: for all of my selfhosted applications.

The applications connect to the TrueNAS’ storage via NFS. I have two identical HDDs as a mirror, another one that has no failsafe (but it’s fine, because the data it contains is non-critical), and an external HDD that I want to use for replication, or some other use I still haven’t decided.

Now, the issue is the following. I’ve noticed that TrueNAS complains that the HDDs are Unhealthy and has complained about checksum errors. It also turns out that it can’t run S.M.A.R.T. checks, because instead of using an HBA, I’m directly passing the entire HDDs by ID to the VM. I’ve read recently that it’s discouraged to pass virtualized disks to TrueNAS, as data corruption can occur. And lately I was having trouble with a selfhosted instance of gitea, where data (apparently) got corrupted, and git was throwing errors when you tried to fetch or pull. I don’t know if this is related or not.

Now the thing is, I have a very limited budget, so I’m not keen on buying a dedicated HBA just out of a hunch. Is it really needed?

I mean, I know I could run TrueNAS directly, instead of using Proxmox, but I’ve found TrueNAS to be a pretty crappy Hypervisor (IMHO) in the past.

My main goal is to be able to manage the data that is used in selfhosted applications separately. For example, I want to be able to access Nextcloud’s files, even if the docker instance is broken. But maybe this is just an irrational fear, and I should instead backup the entire docker instances and hope for the best, or maybe I’m just misunderstanding how this works.

In any case, I have some data that I want to store and want to reliably archive, and I don’t want the docker apps to have too much control over it. That’s why I went with the current approach. It has also allowed for very granular control. But it’s also a bit more cumbersome, as everytime I want to selfhost a new app, I need to configure datasets, permissions and mounting of NFS shares.

Is there a simpler approach to all this? Or should I just buy an HBA and continue with things as they are? If so, which one should I buy (considering a very limited budget)?

I’m thankful for any advice you can give and for your time. Have a nice day!

source

Comments

Sort:hotnew top

possiblylinux127@lemmy.zip ⁨1⁩ ⁨day⁩ ago
I hope you have backups

Never should you pass disks though by VM. The disk and SMART data is not passed though so ZFS has no way of knowing disk status. It also has more overhead than just passing though a PCIe device.

I would strongly recommend that you fix this ASAP

source
- thelemonalex@lemmy.world ⁨1⁩ ⁨day⁩ ago
  Yes, that’s why I’ve posted this question, and I immediately powered the entire NAS off, as to avoid any damage. It’s currently still powered off, until I find the best way to move forward. What I’m afraid of is that if I try to import the pools that were managed by the TrueNAS VM into a bare metal TrueNAS install, or Proxmox, that it won’t work correctly, or that I could lose data.
  
  source
  - uncloaked@sh.itjust.works ⁨6⁩ ⁨hours⁩ ago
    This might be a late reply but if your motherboard supports iommu you can pass your motherboard’s sata controller as a pcie device that is how my current TrueNAS VM is setup
    
    source
    -> View More Comments
  - possiblylinux127@lemmy.zip ⁨1⁩ ⁨day⁩ ago
    I would just pickup a simple PCIe sata card. They are fairly inexpensive and are idea for what you are doing.
    
    Going forward, make sure you keep backups of important data.
    
    source
    -> View More Comments
MangoPenguin@lemmy.blahaj.zone ⁨1⁩ ⁨day⁩ ago
Proxmox supports ZFS natively with management in the WebUI. So you could get rid of TrueNAS entirely and not need to deal with HBA pass-through or anything.

You also wouldn’t need NFS or have to deal with shares, as the data is available directly to Proxmox Containers via bind mounts.

source
- thelemonalex@lemmy.world ⁨1⁩ ⁨day⁩ ago
  Okay, if Proxmox can handle all that, I’ll be glad to ditch TrueNAS. However, I’m afraid that I won’t know how to migrate. I’ve found this reddit thread about someone who tried to do the same thing (I think) and accidentally corrupted their pools. About skipping NFS shares, that would be a big improvement for me, but I’m very unfamiliar with bind mounts. If I understand correctly, you can specify directories that live on the Proxmox Host, and they appear inside the VM, right? How does this compare to using virtual storage? Also, how can I replicate the ZFS pools to an external machine? In any case, thank you for that info!
  
  source
  - MangoPenguin@lemmy.blahaj.zone ⁨1⁩ ⁨day⁩ ago
    Migration should be as simple as importing the existing ZFS stuff into the Proxmox OS.
    
    If I understand correctly, you can specify directories that live on the Proxmox Host, and they appear inside the VM, right?
    
    source
    -> View More Comments
3dcadmin@lemmy.relayeasy.com ⁨1⁩ ⁨day⁩ ago
Debian supports zfs so why the extra hassle of truenas? Seems to be a lot of extra work to add a vm when you could just use zfs in Debian. Or install something like Minio in Debian and use that to manage the data in s3 style buckets, again no vm needed

source
- thelemonalex@lemmy.world ⁨1⁩ ⁨day⁩ ago
  Okay, no VM, understood, but I do want to use a GUI for ZFS, because I’m basically a noob.
  
  source
  - Lem453@lemmy.ca ⁨1⁩ ⁨day⁩ ago
    One thing to consider is that once zfs is setup there really is no significant intervention that is needed. I probably haven’t done anything to my proxmox zfs array in years.
    
    I know its almost a meme to say just learn command line, but unfortunately in this case it will really help you understand what is happening and it also just takes a few commands to setup up once and then never worry about it again.
    
    After the inital setup, the zfs GUI will be pretty much unused.
    
    For commands I don’t use often, i use a note taking software to keep track of commands I used during setup because years go by before I use it again. I find the GUI often changes in that time making it harder to replicate whereas command line is the same and easier to document.
    
    source
    -> View More Comments
  - 3dcadmin@lemmy.relayeasy.com ⁨1⁩ ⁨day⁩ ago
    Well erm that I can’t get around… but I guess it depends on what you want to self host. I use truenas as a storage solution, standalone and it works very well for that - I have had it up and running for years and years. It also stores personal files and is not exposed to the internet apart from updates and a few other things. Then I have an old 4-node server from Supermicro which hosts the stuff I need and uses truenas to store the data. Now that is probably overkill but the data is safe and backed up well. Truenas has apps for a lot of things, and other ways to host docker containers if you have the latest scale community edition. So perhaps you could do it that way
    
    source
    -> View More Comments
Cyber@feddit.uk ⁨2⁩ ⁨days⁩ ago
You should have all your data separately stored, it shouldn’t be locked inside containers, and using a VM hosted on a device to serve the data is a little convoluted

I personally don’t like TrueNAS - I’m not a hater, it just doesn’t float my boat (but I suspect someone will rage-downvote me 😉)

So, as an alternative approach, have a look at OpenMediaVault

It’s basically a Debian based NAS designed for DIY systems, which serves the local drives but it also has docker on, so feels like it might be a better fit for you.

source
- thelemonalex@lemmy.world ⁨1⁩ ⁨day⁩ ago
  I tried OMV in the past, but I found TrueNAS to be more intuitive… but that’s just personal preference I guess, and I’m not opposed to using OMV. Are you suggesting, then, that I run OMV on bare metal, and use it for everything? Or should it be inside a VM? If it’s the former, how easy is it to setup docker, because I’m not that familiar with OMV (it’s been a long time since I last checked it out). Is it like installing it in Debian directly? How does it handle the storage?
  
  source
  - Cyber@feddit.uk ⁨1⁩ ⁨day⁩ ago
    I always prefer bare metal for the core NAS functionality. There’s no benefit in adding a hypervisor layer just to create an NFS / SMB / iSCSI share
    
    OMV comes with it’s own bare metal installer, based on Debian, so it’s as stable as a rock.
    
    If you’ve used it before, you’re probably aware that it needs it’s own drive to install on, then everything else is the bulk storage pool… I’ve used various USB / mSATA / M.2 drives over the years and found it’s a really good way to segregate things.
    
    I stopped using OMV when - IMO - “core” functions I was using (ie syncthing) became containers, because I have no use for that level of abstraction (but it’s less work for the OMV dev to maintain addons, so fair enough)
    
    So, you don’t have to install docker, OMV automatically handles it for you.
    
    How much OMV’s moved on, I don’t know, but I thought it would simplify your setup.
    
    source
    -> View More Comments
rumba@lemmy.zip ⁨1⁩ ⁨day⁩ ago
I would probably shy away from passing the raw disk. There are a few dozen ways to skin that, But in the end I would probably just mount the disc through NFS,smb, whatever it takes. Reading that smart data is paramount for your situation. You could have a bad cable and never know it.

You could run a couple of VMs K8S and longhorn the data, It’s capable of backing up to an S3 compliant storage.

For my home stuff at the moment I’m running unraid with a BTRFS and a parity disc. The first every month I run a scrub if I had any corruption it would find it for me and alert me. It’s slow as balls but more protection is better than less. You can also buy some recycled discs and keep a offline store. I don’t love the recycled discs but for backup purposes that aren’t running 24x7 they’re better than nothing.

source
- thelemonalex@lemmy.world ⁨1⁩ ⁨day⁩ ago
  The problem is that I’ve never worked with Kubernetes, and it wouldn’t allow me to port over my current docker instances, afaik. And I’m not ready to pay for Unraid, I’m sorry, I need to set up the server as frugally as possible. Still, thank you for the info, I’ll keep it in mind, while I decide how to move forward.
  
  source
  - rumba@lemmy.zip ⁨1⁩ ⁨day⁩ ago
    
    I’ll keep it in mind, while I decide how to move forward. How I do it isn’t necessarily ideal. just throwing out options.
    
    The chances of corruption from passing virtualized disks in truenas is lower than passing raw disks. I’d move them to virtualized disks.
    
    As far as your SMART data, make a ventoy+debian live usb, boot your host off that and install smartmontools and use smartctl to read your SMART data. You host should still have access to that data. If you have smart errors, you’ll need to fix those first.
    
    source
IronKrill@lemmy.ca ⁨1⁩ ⁨day⁩ ago

I have two identical HDDs as a mirror, another one that has no failsafe (but it’s fine, because the data it contains is non-critical)

On separate pools, I hope? My understanding of ZFS is that the loss of any vdev will mean the loss of the pool, so your striped vdev should be in its own pool that you don’t mind losing.

source
- thelemonalex@lemmy.world ⁨1⁩ ⁨day⁩ ago
  Correct, they’re two separate pools, and the stripe one doesn’t contain any valuable data.
  
  source
frongt@lemmy.zip ⁨2⁩ ⁨days⁩ ago
If you pass a whole raw disk, not virtualized, then TrueNAS should not complain. I don’t know if you can do that in proxmox, I haven’t tried.

Personally I’d get rid of TrueNAS. Even if docker is down, the VM with the data is still up and accessible over anything running on the VM, like scp via ssh.

source
- non_burglar@lemmy.world ⁨1⁩ ⁨day⁩ ago
  
  If you pass a whole raw disk, not virtualized, then TrueNAS should not complain.
  
  Although Smart data counters live on each individual disk, it is accessed via the disk controller interface. No controller, no smart data.
  
  source
- possiblylinux127@lemmy.zip ⁨1⁩ ⁨day⁩ ago
  If you pass the disk the meta data isn’t passed though including SMART data so it is a ticking time bomb.
  
  source
- thelemonalex@lemmy.world ⁨2⁩ ⁨days⁩ ago
  I passed the whole raw disk in theory, but I’ve read that even if you do that, it’s still virtualized somehow. But I don’t know enough about it, I’m just basing this on the research I’ve done so far. About skipping TrueNAS, maybe you’re right, but if I also want to set up SMB shares? Is there a tool or something that makes it easy for someone with limited experience? It would also need to handle replication of data, and the advantage of ZFS Replication over using something like rsync, is that you can simply push the delta between snapshots, instead of replacing entire files, or at least that’s how I’ve understood this.
  
  source
ikidd@lemmy.world ⁨1⁩ ⁨day⁩ ago
I run a docker host in Proxmox using ZFS datasets for the VM storage for things like my mailserver and NexcloudAIO. When I backup the docker VM, it snapshots the VM at a point in time, and backs up the snapshot to PBS. I’ve restored from that backup and it’s like the machine had just shut down as far as the data is concerned. It journals itself back to a consistent state and no data loss.

I wouldn’t run TrueNAS at all because I have no idea how that’s managing it’s storage and wouldn’t trust the result.

source
- thelemonalex@lemmy.world ⁨1⁩ ⁨day⁩ ago
  Wait, so if I understood correctly, you’re managing the ZFS pools directly in Proxmox, and then you have a VM that’s running docker, and using the storage that is managed by Proxmox, right? Hmm, sounds like a good solution. Is there any documentation or article that you could recommend, so that I can take a closer look? Also, how could I handle SMB shares?
  
  source
  - ikidd@lemmy.world ⁨1⁩ ⁨day⁩ ago
    Yes. So my debian docker host has some datasets attached: Image
    
    mounted via fstab: Image
    
    and I specify that path as the datadir for NCAIO:
    
    Image
    
    Then when PBS calls a backup of that VM, all the datasets that Proxmox is managing for that backup take a snapshot, and that’s what’s backed up to PBS. Since it’s a snapshot, I can backup hourly if I want, and PBS dedups so the backups aren’t using a lot of space.
    
    Other docker containers might have a mount that’s used as a bind mount inside the compose.yml to supply data storage.
    
    Also, I have more than one backup job running on PBS so I have multiple backups, including on removable USB drives that I swap out (I restart the PBS server to change drives so it automounts the ZFS volumes on those removable drives and is ready for the next backup).
    
    You could mount ZFS datasets you create in Proxmox as SMB shares in a sharing VM, and it would be handled the same.
    
    As for documentation, I’ve never really seen any done this way but it seems to work. I’ve done restores of entire container stacks this way, as well as walked the backups to individually restore files from PBS.
    
    If you try it and have any questions, ping me.
    
    source
    -> View More Comments
alexgreen@techhub.social ⁨2⁩ ⁨days⁩ ago
@thelemonalex I usually find that sata cable or connector are bad.
Personally I use proxmox as a host and share bulky nfs mounts for each VM like Immich, Plex/Jellyfin. For gitea and other small VM - use VM virtual drive and back it up periodically
source
- thelemonalex@lemmy.world ⁨2⁩ ⁨days⁩ ago
  So if I understand correctly, you’re managing the storage directly within Proxmox, instead of using a VM for that, right? Are the tools good? Does it support ZFS replication, SMB and things like that?
  
  source
  - alexgreen@techhub.social ⁨2⁩ ⁨days⁩ ago
    @thelemonalex I use native linux nfs server. Manage shares using /etc/exports file.
    For SMB, I use vm with samba and mount a few folders over nfs from proxmox (it just gives me a static IP in case if I decide to change something from proxmox side).
    I manage zfs on the proxmox host. I think you can't export zfs dataset to be managed by VM (except drive passthrough).
    My infra has 1 proxmox bulky node with 2 HDD (zfs) and 3 nodes with small ssd/nvme in the cluster.
    source
    -> View More Comments
philoko@ani.social ⁨2⁩ ⁨days⁩ ago
I was in somewhat of a similar situation but some of the SATA ports were bad and ended up buying a cheap SATA to PCIE card and passing it instead but that still had some issues so I ended up moving TrueNAS to its own dedicated machine and haven’t had any problems since.

Just my two cents.

source
- Nollij@sopuli.xyz ⁨2⁩ ⁨days⁩ ago
  To anyone reading, do NOT get a PCIe SATA card. Everything on the market is absolute crap that will make your life miserable.
  
  Instead, get a used PCIe SAS card, preferably based on LSi. These should run about $50, and you may (depending on the model) need a $20 cable to connect it to SATA devices.
  
  source
  - possiblylinux127@lemmy.zip ⁨1⁩ ⁨day⁩ ago
    I have a cheap PCIe card I bought and it works fine.
    
    It cost like 10$ and has been rock solid. What is the issue?
    
    source
    -> View More Comments
  - thelemonalex@lemmy.world ⁨2⁩ ⁨days⁩ ago
    Are there any specific limitations/requirements? Any recommended models or things to look out for? I looked on Amazon, and they range from around $30 to $200, and I really have no criteria, and I want to spend as little as possible.
    
    source
    -> View More Comments
- thelemonalex@lemmy.world ⁨2⁩ ⁨days⁩ ago
  Thank you, that’s actually very helpful, because I’m also kinda concerned that the HBA (or SATA to PCIE card) solution won’t fix this issue entirely, and it’s also relatively old system, and I’m worried about compatibility. Having two machines though is also kind of out of the question, because I don’t want to use that much electricity, it’s pretty expensive here where I live. I do have a Raspberry Pi, which could be used for some selfhosted services, but not the demanding ones, and not running alongside each other. I mean, my current PC sometimes struggles already, so I guess the Raspberry Pi will perform even worse… Maybe I can find a solution that allows to still manage storage reliably and flexibly, and also be able to have docker containers.
  
  source
  - philoko@ani.social ⁨1⁩ ⁨day⁩ ago
    TrueNAS scale is the Linux version of TrueNAS with docker support. Might be worth looking into
    
    source
    -> View More Comments
carzian@lemmy.ml ⁨2⁩ ⁨days⁩ ago
If you’re only doing a VM or two, I’d get rid of proxmox and run truenas directly. It’s gotten better for VMs.

Also make sure you read up on the ecc requirements for truenas if you’re not using ecc ram

source
- thelemonalex@lemmy.world ⁨2⁩ ⁨days⁩ ago
  And is it easy for the docker instances inside the VM to access the host’s datasets? About ECC, thank you for bringing it up, because I actually have no idea on the subject, and I’m sure that my current ram isn’t ECC. I’ll look into it. It could explain the issue I had with gitea, right?
  
  source
BCsven@lemmy.ca ⁨2⁩ ⁨days⁩ ago
Send your question to the podcasters at 2.5admins, this seems right up their alley.

source
- thelemonalex@lemmy.world ⁨2⁩ ⁨days⁩ ago
  Thank you, I might raise the issue there, if I struggle to fix the issue. I didn’t know the podcasters, but now I’ll try and listen to a few episodes, and maybe I can continue learning. Thank you for the suggestion
  
  source
  - non_burglar@lemmy.world ⁨1⁩ ⁨day⁩ ago
    They’re gonna say the same thing you’ve read here, which is that if you’re going to virtualize TrueNAS, pass through the controller, not just the disks.
    
    source
  - BCsven@lemmy.ca ⁨1⁩ ⁨day⁩ ago
    They always do Free consulting on air, so if your issue gets to them and they have a spot, you get a very very competant answer
    
    source