> "your usage will be measured in tokens, which we do not define, nor give you a way to measure them
They have an open source tokenizer on GitHub, they indicate which tokenizer they use per model, and they have a simple to use website that tokenize text and visualize it.
They have an open source tokenizer on GitHub, they indicate which tokenizer they use per model, and they have a simple to use website that tokenize text and visualize it.