I’m in the same boat as well, for some reason my music apps can’t seem to see my Google smart speakers, but strangely they can see my Nvidia shield and cast to that.

Looked through my app permissions I honestly can’t see anything out of the ordinary or plainly obvious. Both my speakers and android TV (shield) are on separate segment VLAN.

As for granting the permission, I generally like to grant on a app by app basis, lots of apps ask for internet access as a example but don’t actually require it to function. All my game apps for example are completely offline.