Related quote from Karpathy:
Tokenization is at the heart of much weirdness of LLMs. Do not brush it off.
• Why can't LLM spell words? Tokenization.
• Why can't LLM do super simple string processing tasks like reversing a string? Tokenization.
• Why is LLM worse at non-English languages (e.g. Japanese)? Tokenization.
• Why is LLM bad at simple arithmetic? Tokenization.
• Why did GPT-2 have more than necessary trouble coding in Python? Tokenization.
• Why did my LLM abruptly halt when it sees the string "<|endoftext|>"? Tokenization.
• What is this weird warning I get about a "trailing whitespace"? Tokenization.
• Why the LLM break if I ask it about "SolidGoldMagikarp"? Tokenization.
• Why should I prefer to use YAML over JSON with LLMs? Tokenization.
• Why is LLM not actually end-to-end language modeling? Tokenization.
• What is the real root of suffering? Tokenization.
Tokenization is at the heart of much weirdness of LLMs. Do not brush it off.
• Why can't LLM spell words? Tokenization.
• Why can't LLM do super simple string processing tasks like reversing a string? Tokenization.
• Why is LLM worse at non-English languages (e.g. Japanese)? Tokenization.
• Why is LLM bad at simple arithmetic? Tokenization.
• Why did GPT-2 have more than necessary trouble coding in Python? Tokenization.
• Why did my LLM abruptly halt when it sees the string "<|endoftext|>"? Tokenization.
• What is this weird warning I get about a "trailing whitespace"? Tokenization.
• Why the LLM break if I ask it about "SolidGoldMagikarp"? Tokenization.
• Why should I prefer to use YAML over JSON with LLMs? Tokenization.
• Why is LLM not actually end-to-end language modeling? Tokenization.
• What is the real root of suffering? Tokenization.