That phrasing (at least without the "to you") is fine when you are genuinely open to saying more. It's not fine as a way to refuse to say more. So you're talking about a completely different context.
If they get aggressive and you then decide that you don't want to explain, that's fine. But that's not what happened here.
If they get aggressive and you then decide that you don't want to explain, that's fine. But that's not what happened here.