I think that’s on purpose - it lets you change where your media is playing but keeps Teams set to the headset. The idea is that if you’re listening to something on a speaker and a call comes in, you want to be able to answer without scrambling to change the device first. Especially if you’re already having to take time to pause your media first.