But the Tower of Hanoi can be solved without "tools" by humans, simply by understanding the problem, thinking about the solution, and writing it out. Having the LLM shell out to a Python example that it "wrote" (or rather, "pasted" since surely a Python solution to the Tower of Hanoi was part of its training set) is akin to a human Googling "program to solve Tower of Hanoi", copy-pasting and running the solution. Yes the LLM has "reasoned" that the solution to the problem is call out to a solution that it "knows" is out there, but that's not really "thinking" about how to solve a problem in the human sense.
What happens when some novel Tower of Hanoi-esque puzzle is presented and there's nothing available in its training set to reference as an executable solution? A human can reason about and present a solution, but an LLM? Ehh...
What happens when some novel Tower of Hanoi-esque puzzle is presented and there's nothing available in its training set to reference as an executable solution? A human can reason about and present a solution, but an LLM? Ehh...