I don’t think there’s a contradiction there, a term being gendered isn’t all-or-nothing. Certainly, some men attracted to men identify as gay, as well as some lesbian women, and even some bisexual folks of any gender. In that way it isn’t exclusively gendered.

But if I say “the gay community”, I’m guessing the image that evokes in your mind leans heavily towards gay men, compared to a phrase like “the LGBTQ+ community”. Even if the speaker means the same thing by those phrases, the listener likely interprets them differently.