Comment by jadermcs

Current tokens are very problematic, they are constructed by frequency, but frequency doesn't capture morphology and syntax that well. If the model learn how to tokenize internally it's way better then forcing an incorrect tokenization.