Comment by bastien2

That really is the solution: stop misusing phone numbers as secure channels.