Ask HN: my startup addresses a small market. How much of a problem is that?

Longtime HNer on a throwaway account. I've just started work on a startup making a web app for a particular type of business. (Sorry about the intentional vagueness.) There's something like 5000 businesses of this type in the U.S. With subscription billing, if the average customer is paying $25/month, I'm looking at a total market size of $1.5 million/year. Granted, maybe there are additional apps I could build in the same domain, and other unknown sources of revenue, but I'm starting to worry about potential for growth.

Am I overthinking this? Have any of you launched companies into markets of similar (or smaller) apparent size? What problems am I going to run into, and what's the best way around them?