It’s because your return journey is closer to the axis of the earth so your action has less torque.