That is the nature of any federated protocol.

E2EE works well enough within rooms and that is likely where private data is to be anyway. As long as you Matrix and assume that everyone can see your Matrix ID and room IDs you’ll be okay.

XMPP isn’t any better in that regard.