Check out the krun runtime for podman, if you want stronger container isolation from the host. It creates a small VM for each container. Gpu passthrough on linux is limited though.