Comment by mountainriver

> We find that even our best models are generally unable to predict the output of a program given a specific input.

I’ve been thinking on this quite a bit, it’s an interesting property of language models in that they are essentially associative lookup machines.

It feels on some level that we need to be combining this kind of reasoning with something like world models.