I don't know the solution to this problem, but I do think that turning it into a Morton's fork ("men are sexist regardless of whether they speak or not") is not it.
Instead of playing semantics by saying that "it is technically sexism" (and I'm not saying I agree with whether it actually is or not), we could choose to at least stop phrasing the situation like that.
Instead of playing semantics by saying that "it is technically sexism" (and I'm not saying I agree with whether it actually is or not), we could choose to at least stop phrasing the situation like that.