Comment by lopuhin

The paper rises a great point on tokenization affecting perplexity, that we can't compare perplexities of different tokenizers even re-normalizing taking token counts into account, say BPE vs word tokenization. This example nails it: https://twitter.com/Smerity/status/1192252147598909441