yt-dlp requests a specific format and downloads that in its entirety. When you use the YouTube web site, it dynamically adjusts based on your internet connection. If there’s lag, then it’ll drop you to a lower bitrate. So that’s one possible explanation.

There’s a little gear button on the YouTube page that lets you manually select the quality you want. If you specifically select e.g. 1080p then it won’t drop you down, it’ll just lag if your internet is too slow. I’m not 100% sure if that also applies to audio.