So the thing is, bytes are not actually a more "natural" encoding than tokens. I mean, jesus christ Unicode, right? A lot of what the BPE is doing is just
un-doing the wacky arbitrary standards we use to encode characters into bytes, like, say, merging the 3-4 random bytes used to specify the codepoint of some CJK character.
There are so many possible naturalistic encodings. Waveforms, one-hot encoded discrete articulatory features, motor programs for simulated vocal apparatus, IPA, bitmaps of printed characters (handwritten? averaged across a bunch of fonts?), SVG programs to draw characters, decomposing big CJK chars via Ideographic Description Sequence, treating all unicode codepoints equally (kinda halfway between bytes and tokens)...
I don't think a thorough survey exists.
There are so many possible naturalistic encodings. Waveforms, one-hot encoded discrete articulatory features, motor programs for simulated vocal apparatus, IPA, bitmaps of printed characters (handwritten? averaged across a bunch of fonts?), SVG programs to draw characters, decomposing big CJK chars via Ideographic Description Sequence, treating all unicode codepoints equally (kinda halfway between bytes and tokens)...
I don't think a thorough survey exists.