En dashes, I'll grant you, are pointless. Those can go away.
However, em dashes are a different case. The main reason why it's desirable to use em dashes (beside convention) is for clarity of purpose. The hyphen is already a very overloaded character; they're extensively used to denote ranges and link compound words. Importantly, both of those usages do not correspond to pauses in spoken language. If you're voicing a hyphen you're supposed to barrel on through it. An em dash is much closer to a parenthesis, comma, or semicolon. It's a meaningful break in the sentence, in the way that a hyphen isn't.
Now, if it were up to me I'd choose a different character to replace em dashes (maybe underscores), but that's a separate argument.
However, em dashes are a different case. The main reason why it's desirable to use em dashes (beside convention) is for clarity of purpose. The hyphen is already a very overloaded character; they're extensively used to denote ranges and link compound words. Importantly, both of those usages do not correspond to pauses in spoken language. If you're voicing a hyphen you're supposed to barrel on through it. An em dash is much closer to a parenthesis, comma, or semicolon. It's a meaningful break in the sentence, in the way that a hyphen isn't.
Now, if it were up to me I'd choose a different character to replace em dashes (maybe underscores), but that's a separate argument.