If you were to say “black people are bad” then the rebuttal typically becomes “due to established power structures, white men can’t be oppressed, therefore I can say what I want about them”.

I think it’s hard to have a reasoned discussion where you say “all men do X” because for every X, there’s probably a bunch of exceptions. There is a great amount of variance in male behavior.