A lot of it depends on your distro. I use exclusively Mint and Debian (primarily Debian), and everything works fine on both of those. My laptop runs Debian 13 and has the iGPU and an RTX4070, and one of my servers has both an RTX A6000 and a T400, both being passed through Proxmox into two different Debian 13 VMs. Everything works without issue. Before Debian 13 on the laptop I had Mint 22, and before that Ubuntu 23.10, and both worked without issue as well.