The problem is people don't want to associate with an organization lead by someone they have problems with, and a lot of people have problems with his views/behavior. The struggle of the FSF is difficult as it is. He doesn't have to be actively using power/influence.
edited phrasing