I don’t even care about encryption (except normal TLS). If the intent is that it’s for sharing within the family, generally with people being able to see each others’ stuff, and I also control the server, it doesn’t seem that important.

Circles seems like it might fit much of what you’re after. It’s based on Matrix and is intended for this kind of use case, but I don’t think it’s mature enough to onboard everyone to just yet.