Chiang makes some insightful points, e.g. about what we mean by magic.
Then I come to
> [LLMs] can get better at reproducing patterns found online, but they don’t become capable of actual reasoning; it seems that the problem is fundamental to their architecture.
and wonder how an intelligent person can still think this, can be so absolute about it. What is "actual" reasoning here? If an AI proves a theorem is it only a simulated proof?
Then I come to
> [LLMs] can get better at reproducing patterns found online, but they don’t become capable of actual reasoning; it seems that the problem is fundamental to their architecture.
and wonder how an intelligent person can still think this, can be so absolute about it. What is "actual" reasoning here? If an AI proves a theorem is it only a simulated proof?