Ask HN: Why not go after the passionate users?

Whenever I see VCs talking about the potential market for a startup, they always want a market with a lot of people. I mean, it seems obvious. More people means more money and more room to grow, but why not focus on the passionate users? The users that have a significant pain that needs to be solved and will pay out their ass for it. Aren't these the people that we should be focusing on?