This is pure rhetoric, I can flip the argument:

“You care more about the gender than about my material condition.”

Also, the moment I need to let prevail abstract concepts over my material condition (i.e., caring about “my group” being over represented while I am out of a job) is the moment in which the class unity is broken. Me and those women who are out of a job have so much in common that there is no reason for me to consider us part of two separate groups. That’s the whole point of my argument, I advocate for worker solidarity and I absolutely feel that this attitude is overall harmful for it.