I run Postfix, Dovecot and rspamd on my server. The configuration is here: git.dblsaiko.net/systems/tree/…/polaris

There’s also the Simple NixOS Mailserver project which is an abstraction on top of these and has a few more things. I’ve never used it myself though.

Of course, you also have to set up all the standard email setup like DKIM, DMARC, SPF and so on here.