dcode

> the Web side is largely complete and successful

Perhaps from a C++ perspective, say a monolith with kilobytes of bindings driving a canvas. Yet

> the original purpose of Wasm

includes, as per the charter, "and interoperate gracefully with JavaScript and the Web", which is far from complete and, thanks to WASI and the Component Model, hardly improving. I guess it will remain a mystery how WASI could basically replace something as obviously useful as WebIDL-bindings - that is, until someone figures out.

I guess we are about to find out whether there is substance to the precedents. My bet is on "what can go wrong, will go wrong", even more so on Web scale. Let's hope I'm wrong.

I do understand the desire to switch all languages and systems to one encoding, of course. However, Switching a WTF-16 language to UTF-8 removes previously valid values from strings, then exchanging what's errors/mutation on Component boundaries right now with errors/mutation when using string APIs. Can't be done in a backwards-compatible way, and all these languages have a lot of existing code. If backwards compatibility is a goal (say when using a breadcrumbs mechanism as in Swift), one still ends up with WTF-8 underneath, which maps to WTF-16, but is not UTF-8. Hence why I think it's impossible, because the only way to pull this off is by replacing affected string APIs (and/or accepting that old APIs then throw or mutate). Likewise, I see a possible future where JS adopts breadcrumbs, but then with WTF-8 (and perhaps a well-formedness flag), not guaranteed UTF-8. In your use case, that would yield a fast-path if a string is well-formed, but still with the same old fallback. Plus, of course, that having a systems fast-path implies that there is a corresponding JS-interop slow-path (when using AS).

To me it's not so much a question of what's the best / recommended (for new languages) / most used encoding. It's rather the observation that there are so many popular languages operating on Unicode Code Points, not Unicode Scalar Values, and Wasm said it wants to support these (as) well, incl. integration with JS/Web APIs which already share their semantics. And for all of these, the restriction is unnecessary and trivially avoidable by allowing them to pass strings to each other, or from/to JavaScript, without errors or mutation, while not changing anything for those preferring UTF-8. Looking at this the other way around might be valuable as well: What if one would design a Component Model for any of the affected languages? There, introducing an unnecessary discontinuity would be a design mistake I'd argue.

That's not the issue / impression. The issue is that well-formed UTF-16 is rarely used in practice. All of JavaScript, Java, C#, Dart, Kotlin etc. effectively use WTF-16 for compatibility and performance reasons, and that's what's semantically distinct from UTF-8. These have asymmetric value spaces, so that strings in "legacy" languages would sometimes throw or implicitly mutate synchronously. Mixed systems typically use WTF-8 as the common denominator for this reason, i.e. not UTF-8, but Wasm decided against.

Except that picking the most restrictive option prevents two components that are using the same less restrictive semantics to communicate with each other securely (without throwing exceptions or silently mutating data), e.g. Java <-> Java, AssemblyScript <-> JavaScript. To illustrate, if one would design a Component Model for Java, restricting to UTF-8 would make the Component Model for Java a hazard for Java. The same effect happens in a multi-language Component Model, where some languages then work and others don't. Hence "pick the best option" falls short. The argument in all these questionably stonewalled discussions is basically to allow these languages and use cases to exist, which could be as trivial as to make UTF-8 the default if the WASI folks so wish, but also have a Boolean flag for "don't eagerly mutate on WTF-16 pass-through". Yet, even though trivial and rather obvious, this has been fought relentlessly since 2017, and surely one has to wonder why this vehemence.

Despite that speed is not the critical concern (asymmetric value spaces are), UTF-8 must guard against invalid byte sequences, while WTF-16, where all possible values are valid as long as byte length is a multiple of 2, does not. In practice, the guard in UTF-8 is part of the copy loop over a boundary, typically from untyped memory to untyped memory, while WTF-16 can indeed just memcpy. Unless SIMD can be utilized, the difference is about that of a loop over a load into a branch vs. a memcpy, in case this helps to quantify. Expect an additional final memcpy if the UTF-8 copy should fail before storing anything to the receiver's memory.

Well, I tried. Anyway, even if I was the abhorrent monster you keep painting in your almost exclusively ad hominem argumentation while accusing me of what you are undoubtedly guilty of yourself, I'd argue that none of this justifies plain ignoring technical concerns in a standardization effort. This exchange is an almost perfect reflection of the practices prevalent in the Wasm CG, that made the Component Model, and likely other things elsewhere, possible almost uncontested. And surely this is deliberate abuse, and I hope people can see that. And coincidentally, that's exactly my critique. I hope nobody is surprised that being at the receiving end of this, despite your best efforts, for years, is an extraordinarily frustrating experience, and that this is exactly the point.

Allow me to focus on the technical arguments, that I think fall short. Sure, one can go and make a second string type, or a bunch of throwing APIs. I'd question that this actually improves a language in a tangible way, or that anyone would want that for a rational reason that is not induced from the outside. When not doing so, it's incompatible in the sense of the word. Whether incompatibility isn't much of a deal depends, I guess. For someone not having a respective use case, perhaps, but for someone else having exactly that use case, say hashing substrings to then discover unexpected collisions, or streaming 1K chunks of strings over component boundaries, then discovering mojibake after concatenating, it might very well be significant, or even expensive. I mean, there are good reasons all those languages try the best they can to prevent that in their native habitats. What amazes me is that all that came to be because someone has formulated a desire, that could easily be fulfilled in addition with a boolean flag for W/UTF, but refuses to include such a trivial compromise, which surprisingly has more weight than any evidence, or precedents like WebIDL, JSON, or the various language standards. I find this highly concerning, since it conflicts with my understanding of responsible engineering. Also conflicts with Wasm's communication, that literally states that Wasm executes in the same semantic universe as JavaScript, and maintains backwards-compatibility with the Web. Perhaps, if there is something fruitful to spin a narrative around, then that these decisions undermine the exact value proposition of AssemblyScript, that was supposed to be used in tandem/closely with JavaScript, which now becomes risky on the fundamental level of the most prominent higher-level data type, strings. Plus, of course, when two AssemblyScript components communicate. That makes these string decisions particularly unfortunate for me personally after having spent all that time and effort, working towards Wasm's goals in good faith, perhaps explaining my persistence on the matter. Quite a dilemma.

AssemblyScript maintainer here. Can confirm, other language communities have practically zero influence on the design. In fact, AS was present during "specification" and has been completely annihilated for disagreeing with what the chairing majority has come up with there. It's particularly ironic that the Component Model suddenly targets Web integration again, or that this is now coined "language neutral". In relation, the syntax is more of a minor detail.