I think the main problem is that SMS sender numbers can be easily spoofed (might depend on country, operator, …), so relying on “this message came from where it says it came from” is not really possible.
It might not be an issue for some types of usage, but sounds risky if used for account security/recovery/etc.
It might not be an issue for some types of usage, but sounds risky if used for account security/recovery/etc.