Uh, they don’t need to understand how it works to recognize how it does not work.
> A common definition of a rational agent (human or artificial), which the authors adopted, is whether it reasons according to the rules of logic and probability.
Assuming that definition then LLMs clearly do not start from those two first principles; it is manifestly obvious that the rules of logic are not programmed into them explicitly, and since you acknowledge that LLMs cannot cope with mathematical operations (the underpinning of said rules) they also cannot have discovered either the rules of logic or probability implicitly.
Again, if relying solely on that definition (whatever the value of doing so) then it’s perfectly rational to label LLMs as irrational.
> A common definition of a rational agent (human or artificial), which the authors adopted, is whether it reasons according to the rules of logic and probability.
Assuming that definition then LLMs clearly do not start from those two first principles; it is manifestly obvious that the rules of logic are not programmed into them explicitly, and since you acknowledge that LLMs cannot cope with mathematical operations (the underpinning of said rules) they also cannot have discovered either the rules of logic or probability implicitly.
Again, if relying solely on that definition (whatever the value of doing so) then it’s perfectly rational to label LLMs as irrational.