Comment Comment by RC_ITR In practice, GPT uses byte-pair encoding [0] for each Unicode character.That’s why cases are treated differently - they’re different in Unicode.This is also the only way to teach a model how to properly capitalize things (since there are no human defined rules).[0] https://towardsdatascience.com/byte-pair-encoding-subword-ba.... parent RC_ITR 3 years ago
That’s why cases are treated differently - they’re different in Unicode.
This is also the only way to teach a model how to properly capitalize things (since there are no human defined rules).
[0] https://towardsdatascience.com/byte-pair-encoding-subword-ba....