But you lose the semantics you choose to not care about. BPE is equally equipped to not fuse single chars if they have downstream value. Most modern tokenizers are already trained with
the target text corpus in mind.
For example Google's propabilistic Unigram embedding model is trained jointly with the end task, so if you would train it on a DNA set where individual chars matter it probably would not fuse them
So the interesting question is rather how the authors manage to maintain perf on all modalities/tasks with a single corpus or if they are benchmaxxing something.
For example Google's propabilistic Unigram embedding model is trained jointly with the end task, so if you would train it on a DNA set where individual chars matter it probably would not fuse them
So the interesting question is rather how the authors manage to maintain perf on all modalities/tasks with a single corpus or if they are benchmaxxing something.