Putting aside the question of legality, a port scan that does not have prior authorization is pretty generally considered "impolite". As an analogy, it's certainly not illegal to touch someone else's laptop screen, and if I gesture at something and accidentally land a fingerprint on someone's screen, I'll apologize. Legality notwithstanding, it's kind of a dick move if you see me do that, to walk over and tell me I don't need to apologize to the person, and then plant your own fingersmudge right there next to mine.
In any case, if you're located inside the US, it's almost certainly a violation of your ISP's terms of service.
In any case, if you're located inside the US, it's almost certainly a violation of your ISP's terms of service.