tptacek nailed it with regards to it being historically racial, but I've also seen, to your point, animosity towards anything that isn't a "traditional" or nuclear family. Fear of the different and unknown is a powerful motivator of the people who bring these complaints. Also animosity towards poverty and the belief or perception of material increases in crime and/or traffic.
(property owner, many units, yimby, have to interface with the citizenry at zoning meetings, etc)
(property owner, many units, yimby, have to interface with the citizenry at zoning meetings, etc)