I suppose you’re right, but forging that kind of thing would be difficult, also considering the PKI already in place. If someone has their own email server and they sign/encrypt their email, and host their public key on a key server somewhere, it’s highly unlikely that all three would be compromised. and even if that fails, you could just meet up with them and exchange flash drives with keys.