Why would they use more energy in one direction versus another? This doesn’t really make sense to me. Heating and cooling is just swapping which element is the condenser and which element is the evaporator.

The gradient determines that. Moving heat energy from inside ambient 25°C to outside ambient 30°C is easier than moving heat energy from outside ambient 5°C to inside ambient 20°C, for example.