I switched from Dockerd to K3s. First you get the benefits of the Kubernetes API but also Pod Security Context, Pod Security Admission and Network Policies which help to reduce attack surface while simplifying your setup. But if you do want to use Podman look into running your containers as read only, drop all capabilities and unprivileged.