Comment by JimDabell

Is this a reimplementation of Fly.io’s Tokenizer? How does it compare?

https://fly.io/blog/tokenized-tokens/

https://github.com/superfly/tokenizer