You may be able to get more granularity by decoupling Google TV volume from the TV’s own volume. To do this, you need to disable HDMI CEC control over volume in the remote’s settings.