I’m going to add to the sun point, and remark that the cool range of tones is associated with snow and ice, since these reflect the blue of the sky. So even if people didn’t need fire to cook or live their lives at all, they would associate the blue ranges with winter.

Water in its liquid state also looks blue, and is usually associated with something refreshing in warm areas. So even in the tropics you would have a reason for associating blues with cools.

In contrast, most desert areas are naturally in orange ranges. You can also argue autumn forests would look orange and they’re not exactly warm, but the light in an autumn forest doesn’t bounce as much or as tinged compared to light conditions in desert areas, where the soil or sand is yellow or orange -red, and where whatever casting a shadow is probably the ground itself.

Finally, human bodies are more in a warm range tone (no matter the skin color) when compared to the same body suffering cold. Nails and lips take on a bluish hue, so do fingertips.

I think you would need to tweak your universe a little bit more to achieve a reversal in color association.