The problem is that if that is your threat model, then the VPS provider, ISP, and literally everything between you and letsencrypt can pull a conpromised key fro letsencrypt.

This actually happened btw, an xmpp server was attacked this way, they compromised not the server itself, but the VPS provider MITMed their traffic: www.devever.net/~hl/xmpp-incident

If your threat model involves this, then the only solution is Tor, which eliminates these requirements of trust.