This is a great review but it still misses what seems like the point to me: these models don't do any actual reasoning. They're doing the same thing that DALL-E etc. does with images: using a superhuman store of potential outcomes to mimic an outcome that the person entering the prompt would then click a thumbs-up icon on in a training model.
Asking why the model doesn't explain how the code it generated works is like asking a child who just said their first curse word what it means. The model and child alike don't know or care, they just know how people react to it.
Asking why the model doesn't explain how the code it generated works is like asking a child who just said their first curse word what it means. The model and child alike don't know or care, they just know how people react to it.