Comment

Comment by Toxygene

Another option would be to detect and/or normalize Unicode input using the recommendations from the Unicode consortium.

https://www.unicode.org/reports/tr39/

Here's the relevant bit from the doc:

> For an input string X, define skeleton(X) to be the following transformation on the string:

    Convert X to NFD format, as described in [UAX15].
    Remove any characters in X that have the property Default_Ignorable_Code_Point.
    Concatenate the prototypes for each character in X according to the specified data, producing a string of exemplar characters.
    Reapply NFD.
The strings X and Y are defined to be confusable if and only if skeleton(X) = skeleton(Y). This is abbreviated as X ≅ Y.

This is obviously talking about comparing two string to see if they are "confusable" but if you just run the skeleton function on a string, you get a "normalize" version of it.

parent Toxygene 2 years ago

Discussion

Live from the official Hacker News API

A modern reader for Hacker News · SvelteKit + Skeleton