Comment Comment by minimaxir It's worth noting that OpenAI mixed it up and is not using the GPT-2 tokenizer this time.It's unclear what tokenizer they are using and the documentation is being coy about it. It could be a more efficient or a less efficient tokenizer. parent minimaxir 3 years ago
It's unclear what tokenizer they are using and the documentation is being coy about it. It could be a more efficient or a less efficient tokenizer.