An LLM is just a brain in a vat.
Our brains would also not reason in such a condition.
Given the right framework that can do miraculous things.
I am not saying that these systems are conscious, but they are able to abstract their context in a way allowing them to understand their own limitations.
https://www.finextra.com/blogposting/31255/adaptability-as-e...