I don't know where you're getting your data but the only rigorous analysis I've ever seen of political polarization is based on DW-NOMINATE scores, and it does indeed confirm that it's the right wing that's been drifting right.