Cool stuff. From my understanding, this is less valuable at inference time and more useful when running offline pre-training data prep.
When tokenizing terabytes of text for your training corpus, the speedup here is probably doing real work in saving you time (and money?). You get a faster iteration cycle when figuring out and adjusting your datasets.
When tokenizing terabytes of text for your training corpus, the speedup here is probably doing real work in saving you time (and money?). You get a faster iteration cycle when figuring out and adjusting your datasets.