My recommendation would be, have two machines: new for all your services, and use the old hw for your NAS. Each could be whatever OS you’re comfortable with using. Most everything on the services machine could be in docker configs, including network mount points to the NAS. You might be able to get away with using the 1080TI in the services box depending on what all you want to do (AI stuff, or newer stream transcoding requirements may require newer hw).

Moving the data from the old NAS to a new one without new disks will be a challenge, yes.

I have a TrueNAS box and used jails for services. I recently set up a debian box separately, and am switching from jails on truenas to docker on debian. Wish I had done this from the start.