blochist
Joined 1 year ago
Karma
32
Submissions
13
Member since
Mar 4
Joined 1 year ago
Basically, OpenAI is trained on web data that has a number of words where splitting on -der makes sense (e.g., mur-der, un-derstanding, won-derful; although the most common occurrence in a GitHub search for "\\xadder" is what appears to be an incorrectly encoded string "L\xc3\xadder", probably from the Portuguese and Spanish "Lí-der").
Anyways, using the o200k tokenizer `mur\xadder` yields two tokens (88762 and 179582). 88762 encodes "mur" and 179582 encodes "\xadder".