I use Jellyfin. I think in your use case, each user would be setup have their own collection. You can enable or disable collections on a per user basis as will as a per client basis.

Downside is that the default web interface isn’t great as a music player. It does the job but it’s not great.
Other hand, multiple music-first clients exist for a lot of different platforms. Odds are good you can find a client that suits how you listen to music.