Comment Comment by omneity I’m working on this problem using a vocab-free, byte-based approach. It’s definitely solvable.https://huggingface.co/posts/omarkamali/593639295164067https://huggingface.co/blog/omarkamali/tokenization parent omneity 1 month ago
https://huggingface.co/posts/omarkamali/593639295164067
https://huggingface.co/blog/omarkamali/tokenization