I’m guessing it’s because most women carry their phone in a bag, so the bigger phone isn’t inconvenient and has the advantage of the bigger screen.
And I suppose most men prefer the bigger screen size, and they are convenient enough in the available sizes. I use a 6.7 inch, and it fits fine in a pocket for me.
Also note that although we have way bigger screens on modern phones, the bezels are way smaller, on the first smartphones the screen was only about 50% of the front face. So a 10 year old 4 inch phone can be about as big as a new 6 inch.