knight666
Joined 14 years ago
Joined 14 years ago
As you get down the list you'll notice what a pain in the ass the special cases are. There's a special case for the final sigma in a Greek word:
03A3; 03C2; 03A3; 03A3; Final_Sigma; # GREEK CAPITAL LETTER SIGMA
You must remove the dot from "i" when upper or titlecasing... but only in Lithuanian: 0307; 0307; ; ; lt After_Soft_Dotted; # COMBINING DOT ABOVE
Etc. etc. By the way, my implementation for case mapping started out similarly to yours, but I ultimately solved the problem using a binary search in a huge look-up table: https://bitbucket.org/knight666/utf8rewind/src/c22e458912952...Unicode case mapping is just a huge mess of exceptions, but that's more the humans' fault than the standard.
The library currently doesn't employ any kind of fuzzing. I rely on multiple tests for every kind of input. I've found that is a pretty reliable way to test for "unknown unknowns", even if it's extremely time-consuming.
Adding testcase fuzzing is definitely something to consider though, because it would most likely have found the very issues this release fixes.
It's very thoroughly tested but it hasn't been used in any serious projects yet.
However, you are correct about codepoint_read's behavior. I actually have a test for passing NUL to the function, but the value of decoded isn't checked:
It's definitely a bug, but relatively harmless. The function is only used internally and the calling sites guard against that kind of attack.Nevertheless, I thank you for the extra set of eyeballs. I have been looking at these functions far too long, you tend to gloss over obvious problems after a while.
I'll be sure to get a fix in for the next release. :)