I used the desktop drivers as well (on arch from the extra repo) for my headless arch server.

Regarding nvidia container toolkit once it was installed I added this to my Jellyfin docker compose:

deploy: resources: reservations: devices: - driver: nvidia capabilities: [gpu]