Another way to look at it: You already know water expands when it freezes.

So if you freeze 1 litre of water, you’ll get more than one litre of ice. To get exactly 1 litre of ice, you have to take some of the ice away, so you have less stuff -> ice is lighter.