If you understand how LLMs work, you should disregard tests such as:
- How many 'r's are in Strawberry?
- Finding the fourth word of the response
These tests are at odds with the tokenizer and next-word prediction model.
They do not accurately represent an LLM's capabilities.
It's akin to asking a blind person to identify colors.
- How many 'r's are in Strawberry?
- Finding the fourth word of the response
These tests are at odds with the tokenizer and next-word prediction model. They do not accurately represent an LLM's capabilities. It's akin to asking a blind person to identify colors.