But why can't you out build them? If there is demand of ten people who need houses and you build ten and a rich person buys all of them so they can rent them out, build ten more. Now the rich person has to buy those if they want to preserve the rental value of the original ten, so you've doubled their costs. They may not be willing to do this. If they are, build ten more...
At some point the rich person decides it's not worth buying 100 houses so they can collect rent on ten, and then they take a bath when they want to unload the first 90 properties which they can't rent for what they paid, to the benefit of everyone else. And when it's repeatedly demonstrated that it's possible to do this, rich people holding property they're not using will start unloading it and bring prices down without even building anything more.
At some point the rich person decides it's not worth buying 100 houses so they can collect rent on ten, and then they take a bath when they want to unload the first 90 properties which they can't rent for what they paid, to the benefit of everyone else. And when it's repeatedly demonstrated that it's possible to do this, rich people holding property they're not using will start unloading it and bring prices down without even building anything more.