It’s not just random text, they are predicting writings and documents produced by humans. They are “language” models.
Language is used to say things about the world. This means that predicting language extremely well is best done through acquiring an understanding of the world.
Take a rigorous, well-written textbook. Predicting a textbook is like writing a textbook. To write a good textbook, you need to be an expert in the subject. There’s no way around this.
The best language models (eg GPT-4) have some understanding of the world. It isn’t perfect, or even very deep in many ways. It fails in ways that we find quite strange and stupid. It isn’t capable of writing an entire textbook yet.
But there is still a model of the world in there. It wouldn’t be able to do everything it is capable of otherwise.
Language is used to say things about the world. This means that predicting language extremely well is best done through acquiring an understanding of the world.
Take a rigorous, well-written textbook. Predicting a textbook is like writing a textbook. To write a good textbook, you need to be an expert in the subject. There’s no way around this.
The best language models (eg GPT-4) have some understanding of the world. It isn’t perfect, or even very deep in many ways. It fails in ways that we find quite strange and stupid. It isn’t capable of writing an entire textbook yet.
But there is still a model of the world in there. It wouldn’t be able to do everything it is capable of otherwise.