I think it is actually the other way around. You can consider the air inside the balloon to have internal energy from the heat. And additionally you have to make room for the balloon in the atmosphere, so you have removed the atmosphere from the volume the balloon takes, which also needs energy. If you consider both you arrive at the concept of enthalpy (H = U + pV), which is very useful for reactions in the atmosphere as pressure is constant. For this example it is not that useful as outside pressure changes when the balloon rises.

Another way to see it, the pressure has no “real” energy. In a ideal gas, the only energy comes from the kinetic or movement energy of the atoms. Each time a gas molecule is hits the balloon envelope it transfers some momentum. The cumulative effect of the constant collisions is the pressure of the gas. If the balloon is now expanding slowly, each collisions also tranfers some energy, in sum building the work the system has to do to the atmosphere. Leading to a decrease in internal, so “real” energy in the balloon. This corresponds to a decrease in temperature.