If you have trouble with outgoing mails, you can use a hybrid approach.

Receive mails directly to your server but use a mail service to relay your outgoing mails. Configuration for that is very simple in mailcow and there are a few dozen (free) transactional email providers (e.g. Scaleway).

That way you can keep receiving your mails privately and only have to give up some privacy when sending mails.