Unfortunately, the Unix terminal is a gigantic pile of hacks accumulated since the late 1960s. It's a miracle the software stack manages to stay upright at all; adding features and getting the entire ecosystem to support them would be tough. Introducing the concept of an external resource that has to be fetched doesn't seem impossible to me, but there are a lot of issues that would have to be thought through.
Anyway, one nice thing about TrueType/OpenType fonts is that the vector representation is extremely well compressed. SVG or similar would be an efficiency loss. In this light, fonts are actually a fairly decent and practical way to represent vector symbols that will be cached and reused. Maybe a minimal practical solution might be to add an ANSI extension that requests an appropriate font for a particular area of the screen, perhaps with a URL from which a specific webfont can be fetched.
Anyway, one nice thing about TrueType/OpenType fonts is that the vector representation is extremely well compressed. SVG or similar would be an efficiency loss. In this light, fonts are actually a fairly decent and practical way to represent vector symbols that will be cached and reused. Maybe a minimal practical solution might be to add an ANSI extension that requests an appropriate font for a particular area of the screen, perhaps with a URL from which a specific webfont can be fetched.