You could write a script that just restarts your container, make sure unprivileged users cannot edit it, and do one of two things:

1. make a sudoers entry for your unprivileged account to call just that script as a user in the docker group with sudo
2. use setuid on the script to have it execute from the docker group even from other users