I use Plexamp for that, Jellyfin does it too. You can assign libraries per user quite easily.

So for 3 users you might have 4 libraries, one per user then a shared library they all have access to.