I had the same problem: Debian host + official Jellyfin Docker image, all set up according to the official guide, but it would fail to transcode anything.

There was no relevant information about what was wrong in the logs so what I did was:

• Copy the ffmpeg command from the logs.
• docker exec -it into the Jellyfin container.
• run the same ffmpeg command manually so I could see the error directly.

Long story short, because the Nvidia toolkit uses the driver/libraries from the host, the error was that I was missing the library libnvidia-encode1 on the host. After installing that, everything works as it should.