Do your smart switches talk to your HomeAssistant server???

Or does your HomeAssistant server talk to the devices?

It’s probably the latter, and in terms of network security the difference is huge. You can restrict your smart switches to their own, untrusted zone with no outgoing permissions and then give HomeAssistant access to them from its zone.

I would also argue that your personal devices and desktop computers are far more sensitive than your HomeAssistant server.