I think we're saying the same thing?

But separately, yes it is possible for men in general and women in general to behave differently. Sexism is when you judge an individual based on generalizations about gender. So it would still not be right to avoid male case workers just because other male case workers you've worked with were assholes, even if it were true that male case workers are more frequently assholes! Judge people for who they are, not the class they belong to.