Other comments are on the right track. You likely need to get your music tagged correctly. From memory, you want the Artist tag to be A feat. B, then the Album Artist tag to be A.

MusicBrainz Picard is great. Especially if your music comes from formally released albums. My issue was having lots of indie playlists/albums. I solved this by having Various Artists labelled in the Album Artist tag (I think).