Transparent Data Tiering on Linux: Why I chose fanotify over FUSE for HuskHoard

Submitted ⁨⁨2⁩ ⁨days⁩ ago⁩ by ⁨deepthinker@lemmy.world⁩ to ⁨selfhosted@lemmy.world⁩

https://www.huskhoard.com/

Ive been working on an opensource HSM (Hierarchical Storage Management) engine called HuskHoard. Most transparent storage tools on Linux use FUSE, but I found the contextswitching overhead was killing my NVMe performance for hot data. I decided to bypass FUSE and use the fanotify API to intercept file access at the kernel level instead. How it works A background janitor moves cold files to slow storage HDD/Tape/S3. It leaves a sparse husk file on the SSD. When an app tries to read the husk, HuskHoard pauses the process, recalls the data, and resumes. Its written in Rust and licensed under AGPL3.0. Github: github.com/huskhoard/huskhoard Technical Architecture: www.huskhoard.com/blog.html Im curious if anyone else here has experimented with fanotify for storage management? I’d love some technical feedback on the architecture

original

Comments

Sort:hotnew top

prenatal_confusion@feddit.org ⁨1⁩ ⁨day⁩ ago
This project seems like a lot of fun! I will try it

Please consider changing the headline font on the website. It is nothing short of a war crime.

original
- deepthinker@lemmy.world ⁨1⁩ ⁨day⁩ ago
  it is an attention getter! I have a hard time reading it sometimes. perhaps it is ready for a freshen up. Please let me know if you have a chance to try it I could use the feedback.
  
  original
  - prenatal_confusion@feddit.org ⁨1⁩ ⁨day⁩ ago
    Oh its getting my attention alright. In the form of discomfort and anger.
    
    original
randy@lemmy.ca ⁨1⁩ ⁨day⁩ ago
Sounds like it has similar goals to bcache (not bcachefs) and LVM cachinch, except that it operates in userspace instead of at the kernel level. Can you explain what the benefits are of keeping this out of the kernel?

original
- WhyJiffie@sh.itjust.works ⁨1⁩ ⁨day⁩ ago
  doesn’t bcache use its own partition type? so an existing partition cannot be used, and so it cannot benefit from compression that certain filesystems could provide.
  
  original
- deepthinker@lemmy.world ⁨1⁩ ⁨day⁩ ago
  hey thanks for the question. you are right there is some overlap with bcache but the big difference is that bcache is block level and huskhoard is file level. keeping it out of the kernel is mostly about safety and portability. if a kernel module panics it takes down your whole server but if a userspace daemon crashes you just restart it. also kernel dev is a nightmare for things like cloud integration… i can use rclone or complex rust libraries for zstd jump tables in userspace way easier than i could in the kernel. it also lets us be file aware. bcache just sees blocks but we can set policies based on file type or age or owner tags which the kernel doesnt really know about. plus it makes it way easier to move between different linux distros or even nas os like truenas without fighting with dkms or kernel versions. basically i wanted the storage intelligence to be portable and safe so you dont have to be a kernel dev to manage your archive
  
  original
WhyJiffie@sh.itjust.works ⁨1⁩ ⁨day⁩ ago
Oh I wanted to make something like this for quite some time! Thank you!

original
- deepthinker@lemmy.world ⁨19⁩ ⁨hours⁩ ago
  I could use some feedback. Please let me know if you see places it can be improved. thanks for the reply
  
  original