Comment by jakecyr

The first (that I found) tokenizer for open source LLMs. It retrieves config files from HuggingFace and can encode / decode text and tokens.

Was created in an hour and can definitely use some work. Would love contributions and feedback!