English speakers tend to have a vocabulary of 30k words or so depending on what exactly you measure.
GPT-4 has 1.7 trillion parameters.
Of course scaling up is quite unpredictable in what capabilities it gets you, but It's not that much of a stretch to imagine that a GPT-4 sized model would have a reasonably sized vocabulary. Certainly worth testing if you have the resources to train such a thing!
English speakers tend to have a vocabulary of 30k words or so depending on what exactly you measure.
GPT-4 has 1.7 trillion parameters.
Of course scaling up is quite unpredictable in what capabilities it gets you, but It's not that much of a stretch to imagine that a GPT-4 sized model would have a reasonably sized vocabulary. Certainly worth testing if you have the resources to train such a thing!