Linux file system developer: we're severely under-resourced

Submitted ⁨⁨1⁩ ⁨year⁩ ago⁩ by ⁨liori@lemm.ee⁩ to ⁨programming@programming.dev⁩

https://lore.kernel.org/lkml/20230810223942.GG11336@frogsfrogsfrogs/

I’ve said this previously, and I’ll say it again: we’re severely under-resourced. Not just XFS, the whole fsdevel community. As a developer and later a maintainer, I’ve learnt the hard way that there is a very large amount of non-coding work is necessary to build a good filesystem. There’s enough not-really-coding work for several people. Instead, we lean hard on maintainers to do all that work. That might’ve worked acceptably for the first 20 years, but it doesn’t now.

[…]

Dave and I are both burned out. I’m not sure Dave ever got past the 2017 burnout that lead to his resignation. Remarkably, he’s still around. Is this (extended burnout) where I want to be in 2024? 2030? Hell no.

source

Comments

Sort:hotnew top

redcalcium@lemmy.institute ⁨1⁩ ⁨year⁩ ago
Reminds me of this xkcd post:

Image

source
stevecrox@kbin.social ⁨1⁩ ⁨year⁩ ago
The linux kernel is very old school in how it is run and originally a big part of the DevSecOps movement was removing a lot of manual overhead.

Moving on to something like Gitea (codeberg) so you have a decent pull request view is quicker/easier than posting a patch to a mailing list.

The branching model of the kernel is something people write up on paper that looks great (much like Gitflow) but is really time consuming to manage. Moving to feature branch workflow and creating a release branches as part of the release process allows a ton of things to be automated and simplified.

Similarly file systems aren't really device specific, so you could build system tests for them for benchmarking and standard use cases.

Setting up a CI to perform smoke testing and linting, is fairly standard.

The previous two points are really each to trigger when a new branch is created/updated, this means review becomes reduced to checking business logic which makes reviews really quick and easy.

Similarly moving on to a decent issue tracker, Jira's support for Epic's/stories/tasks/capabilities and its linking ability is a huge simplifier for long term planning.

You can do things like define OKR's and then attach Epics to them and Stories/tasks to epics which lets you track progress to goals.

You can use issues the way the linux community currently uses mailing lists.

Combined with a Kanban board for tracking, progress of tickets. You remove a ton of pain.

The issue here is open source issue trackers are missing the key productivity enablers of Jira.

The issue is people, the linux kernel maintainers have been working one way for decades. Getting them to adopt new tools will be heavily resisted, same with changing how they work.

Its like everyone outside, knows a breaking the ABI definition from the sub system implementation would create a far more stable ABI which would solve a bunch of issues and allow change when needed, except no one in the kernel will entertain the idea.

source
- CMahaff@lemmy.world ⁨1⁩ ⁨year⁩ ago
  You offered a lot of suggestions, and I’m sure people will disagree over the specifics, but I think your overall point is excellent and not talked about enough. I wonder if anyone has ever even attempted a survey on the ages of maintainers/contributors? I bet it’s skewing older fast.
  
  Nothing wrong with that of course, especially given the project’s age, complexity, and being written in C - but you’re right, at some point you have to attract new talent - people can’t maintain forever.
  
  I’m a 29 year old developer - I didn’t even know you could do git patches via email until recently. And while it’s super cool, it also sounds kinda terrible, especially at the volume they must be receiving? Their own docs are saying the mailing lists receive some 500 emails per day and I can’t imagine the merge process is fun.
  
  So many doc pages are dedicated to how to submit a patch - which is great that it’s documented, and I’m sure it will always be somewhat complicated for a large project - but it also feels like things that are all automatically handled by newer tools / bots which can automatically enforce style checks, etc.
  
  I guess they could argue that the complicated process acts as a filter to people submitting PRs who don’t know what they are doing, but I’d argue it also shuts out talented engineers who don’t have 40 hours to learn how to submit a patch to a project on top of also learning the kernel and also fixing the bug in question.
  
  From what little I read of their git process, does anyone know if there’s anything preventing the maintainer of a subsystem from setting up a more modern method for receiving patches? As long as the upstream artifact to the kernel has the expected format?
  
  source
- cmeerw@programming.dev ⁨1⁩ ⁨year⁩ ago
  
  and its linking ability is a huge simplifier for long term planning.
  
  What long term planning? Who is going to come up with that plan? Will everyone agree to that plan? Who will be paying for the resources to work on that plan?
  
  Combined with a Kanban board for tracking, progress of tickets. You remove a ton of pain.
  
  I am not seeing how that would help. What are you going to do if there is no progress on something? Fire volunteer X because he didn’t make progress on ticket Y (as he has no interest in ticket Y)?
  
  source
- kitonthenet@kbin.social ⁨1⁩ ⁨year⁩ ago
  
  Its like everyone outside, knows a breaking the ABI definition from the sub system implementation would create a far more stable ABI which would solve a bunch of issues and allow change when needed, except no one in the kernel will entertain the idea.
  
  We also can’t go back to fix SMTP either. Simply saying “we need to break ABI” is not a solution for the same reason, the fear Darrick mentions in the email is justified, it’s the fear of e-mails when you change something that breaks everyone’s widget in a way they don’t expect or care to learn about and demand you fix it for them. It’s all well and good to say that they should simply wash their hands of it, but that’s not an option when your inbox is flooded with automated bug reports that don’t explain the problem
  
  source
  - stevecrox@kbin.social ⁨1⁩ ⁨year⁩ ago
    I am actually arguing for a stable ABI.
    
    The few times I have had to compile out of tree drivers for the linux kernel its usually failed because the ABI has changed.
    
    Each time I have looked into it, I found code churn, e.g. changing an enum to a char (or the other way) or messing with the parameter order.
    
    If I was empire of the world, the linux kernel would be built using conan.io, with device trees pulling down drivers as dependencies.
    
    The Linux ABI Headers would move out into their own seperately managed project. Which is released and managed at its own rate. Subsystem maintainers would have to raise pull requests to change the ABI and changing a parameter from enum to char because you prefer chars wouldn't be good enough.
    
    Each subsystem would be its own "project" and with a logical repository structure (e.g. intel and amd gpu drivers don't share code so why would they be in the same repo?) And built against the appropriate ABI version with each repository released at its own rate.
    
    Unsupported drivers would then be forked into their own repositories. This simplifies depreciation since its external to the supported drivers and doesn't need to be refactored or maintained. If distributions can build them and want to include the driver they can.
    
    Linus job would be to maintain the core kernel, device trees and ABI projects and provide a bill of materials for a selection of linux kernel/abi/drivers version which are supported.
    
    Lastly since every driver is a descrete buildable component, it would make it far easier for distributions to check if the driver is compatible (e.g. change a dependency version and build) with the kernel ABI they are using and provide new drivers with the build.
    
    None of this will ever happen. C/C++ developers loath dependency management and people can ve stringly attached to mono repos for some reason.
    
    source
- liori@lemm.ee ⁨1⁩ ⁨year⁩ ago
  Kernel is not a monolithic application, and you cannot develop it like one. There are tons of actors: independent developers, small support companies (like Collabora), corporations, all with different priorities. There is a large number of independent forks (e.g. for obscure devices), that will never be merged, but need to merge e.g. security patches from the mainline. A single project management tool won’t do, not your typical business grade tracking&reporting tool.
  
  CI is already there. Not a central one—again, distributed across different organizations. Different organizations have different needs for CI, e.g. supporting weird architectures that they need to develop against.
  
  There is a reason Torvalds created git—existing tools just wouldn’t work. There might be a place for a similar revolution regarding a bugtracker…
  
  source
Techmaster@lemm.ee ⁨1⁩ ⁨year⁩ ago
Step 1: Don’t murder your wife

source
- 5714@lemmy.dbzer0.com ⁨1⁩ ⁨year⁩ ago
  This must be a reference to someone, right? Casual femicide sounds a bit off.
  
  source
  - ikidd@lemmy.world ⁨1⁩ ⁨year⁩ ago
    ReiserFS
    
    source
    -> View More Comments
  - JWBananas@kbin.social ⁨1⁩ ⁨year⁩ ago
    https://en.m.wikipedia.org/wiki/ReiserFS
    
    source
- Ragerist@lemmy.world ⁨1⁩ ⁨year⁩ ago
  On the bright side, he should have all the time in the world to code, after going to prison.
  
  source
Sanctus@lemmy.world ⁨1⁩ ⁨year⁩ ago
Is there a link to help? I want to help

source
- torpak@discuss.tchncs.de ⁨1⁩ ⁨year⁩ ago
  kernelnewbies.org
  
  source
  - Sanctus@lemmy.world ⁨1⁩ ⁨year⁩ ago
    Thank you
    
    source
- tsz@lemmy.world ⁨1⁩ ⁨year⁩ ago
  Same.
  
  source
ABotelho@lemmy.ca ⁨1⁩ ⁨year⁩ ago
There are so many filesystems. So. Many.

Have any of them considered actually dropping some and pooling efforts into the more promising ones?

source
- Mindlight@lemmy.world ⁨1⁩ ⁨year⁩ ago
  Isn’t “anyone can fork if a project doesn’t really fit their taste” sort of the curse of open source?
  
  Swallowing your pride, merging into another project and taking a less glamorous role in that project is not as easy as it was to fork when steering your project.
  
  This is generally speaking. I’m definitively not saying any of this is that case with the XFS project.
  
  source
  - severien@lemmy.world ⁨1⁩ ⁨year⁩ ago
    
    Swallowing your pride, merging into another project and taking a less glamorous role in that project is not as easy as it was to fork when steering your project.
    
    I don’t think it’s because of the ego. But if you’re working with other people, you need to do a lot of non-coding (non-fun) things. Align thinking, find compromises, establish and follow processes. Things are easier and more fun hacking alone. No processes to limit you, no one telling you “this doesn’t align with the vision of the project” etc.
    
    source
onlinepersona@programming.dev ⁨1⁩ ⁨year⁩ ago
Boy do I find mailinglists impossible to read.

Ontopic: I appreciate what these people are doing, but were I them, a lousy community would have me quit very quickly or never join the project to begin with. Maybe the should just quit and when shit hits the commercial fan, they’ll either pay to get it fixed, get somebody on it to get it fixed, or move on.

Probably fixing whatever non-dev problems they have to make it a nice community to join wouldn’t hurt either. For me, it’s the lack of time and C code. C is a language I absolutely will not touch. There’s not much worse than it IMO and it’s refreshing to see rust slowly entering the kernel.

source
Dadifer@lemmy.world ⁨1⁩ ⁨year⁩ ago
How can we help spread the load? Can we have volunteer sign ups and shift work?

source
- liori@lemm.ee ⁨1⁩ ⁨year⁩ ago
  I guess the best start would be to have a person to organize volunteers.
  
  source
  - rippersnapper@lemm.ee ⁨1⁩ ⁨year⁩ ago
    I think that might help, asking for volunteers. Segregating them based on their abilities (LinkedIn profile for instance). There are a lot of people who have a couple of hours a week they can spend for such a cause. Please check with your colleagues and see if asking for volunteers is possible.
    
    Donating money is also an option, but in this economy, it’s not sustainable to donate significant sums on a recurring basis for most.
    
    source
    -> View More Comments
- HellAwaits@lemm.ee ⁨1⁩ ⁨year⁩ ago
  Go door to door like the Jehovah Witnesses
  
  source
- Blamemeta@lemm.ee ⁨1⁩ ⁨year⁩ ago
  Pretty much, just file system is boring work
  
  source
  - PoisonedPrisonPanda@discuss.tchncs.de ⁨1⁩ ⁨year⁩ ago
    i think the inner workings of an OS is pretty much awesome.
    
    one has to look into to see the beauty.
    
    source
neanderthal@lemmy.world ⁨1⁩ ⁨year⁩ ago
What kind of non-coding work do you need done?

source
- cmeerw@programming.dev ⁨1⁩ ⁨year⁩ ago
  the email lists:
  
  However, every one of these reports has to be triaged, analyzed, and dealt with.
  
  I don’t think non-coding work means non-technical work - you’ll likely still need to dig deep into the technical details to actually be able to help
  
  There is also a link to docs: add maintainer entry profile for XFS
  
  source
kitonthenet@kbin.social ⁨1⁩ ⁨year⁩ ago
I’m having trouble understanding the email, is this for people to help with fs or xfs or both or are they effectively equal? The linked patch is for XFS which I understand Darrick has quit, is he still on fsdevel? Is this plea for either ?

source
- liori@lemm.ee ⁨1⁩ ⁨year⁩ ago
  The thread is an attempt to merge a new file system, bcachefs. This is a large change, requiring a lot of review from experienced developers, and getting anyone to do this work turned out to be difficult. Darrick here started talking how, in general, all development of file systems in Linux is troubled by lack of manpower.
  
  source
  - kitonthenet@kbin.social ⁨1⁩ ⁨year⁩ ago
    So if this is a plea for help, where should I go to see what I can do?
    
    source