A message would not be going to everyone it would exist on your homeserver and the homeserver of whoever your messaging. Second your private messages would not be public u said it yourself its called encryption if u dont trust the encryption no point using signal in the first place. Third even if it made everything public the only meta data would be that u received a mesaage from someone at time xyz (not who just someone) look at sealed sender for more info on this.