Comment on Helm chart installable solutions?

• scrubbles@poptalk.scrubbles.tech ⁨1⁩ ⁨day⁩ ago

  So you have a classic issue of datastorage on kubernetes. By design, kubernetes is node-agnostic, you simply have a pile of compute resources available. By using your external hard drive you’ve introduced something that must be connected to that node, declaring that your pod must run there and only there, because it’s the only place where your external is attached.

  So you have some decisions to make.

  First, if you want to just get it started, you can do a hostPath volume. In your volumes block you have:

  volumes:
    - name: immich-volume
      hostPath:
        path: /mnt/k3s/immich-media # or whatever your path is
  

  The gotcha is that you can only ever run that pod on the node with that drive attached, so you need a selector on the pod spec.
  You’ll need to label your node with something like kubectl label $yourNodeName anylabelname=true, like kubectl label $yourNodeName localDisk=true Then you can apply a selector to your pod like:

      spec:
        nodeSelector:
          localDisk=true
  

  This gets you going, but remember you’re limited to one node whenever you want data storage.

  For multi-node and true clusters, you need to think about your storage needs. You will have some storage that should be local, like databases and configs. Typically you want those on the local disk attached to the node. Then you may have other media, like large files that are rarely accessed. For this you may want them on a NAS or on a file server. Think about how your data will be laid out, then think about how you may want to grow with it.

  For local data like databases/configs, once you are at 3 nodes, your best bet with k3s is Longhorn. It is a HUGE learning curve, and you will screw up multiple times as a warning, but it’s the best option for managing tiny (<10GB) drives that are spread across your nodes. It manages provisioning and making sure that your pods can access the volumes underneath, without you managing nodes specifically. It’s the best way to abstract away not only compute, but also storage.

  For larger files like media and linux ISOs, then really the best option is NFS or block storage like MinIO. You’ll want a completely separate data storage layer that hosts large files, and then following a guide like this you can enable mounting of NFS shares directly into your pods. This also abstracts away storage, you don’t care what node your pod is running on, just that it connects to this store and has these files available.

  I won’t lie, it’s a huge project. It took about 3 months of tinkering for me to get to a semi-stable state, simply because it’s such a huge jump in infrastructure, but it’s 100% worth it.

  source

  • r0ertel@lemmy.world ⁨1⁩ ⁨day⁩ ago

    Excellent write-up. I had Nextcloud running on K3s with its files on a NAS which were shared with Minio and it worked well. I’m looking into Longhorn, but only have 2 nodes and it wants at least 3. I’m reevaluating my resiliency needs in favour of simplification.

    source

    • scrubbles@poptalk.scrubbles.tech ⁨1⁩ ⁨day⁩ ago

      If you’re only at 2 nodes, then I think host paths with node selectors are what you should go with. That gets you up and running in the short term, but know that the conversion later to something like Longhorn will be a process. (Creating the volumes, then copying all the data over, ensuring correct user access, etc).

      source

      • r0ertel@lemmy.world ⁨1⁩ ⁨day⁩ ago

        I have something similar to host paths with node selectors: an NFS provisioner for PVs. The provisioner is tied to the node with the large disk. It’s not resilient to node outages, but allows me to spread pods across the nodes. For my deployments, I’m preferring to use S3 storage wherever possible.

        source