> we investigate the fragility of mathematical reasoning in these models and demonstrate that their performance significantly deteriorates as the number of clauses in a question increases. We hypothesize that this decline is due to the fact that current LLMs are not capable of genuine logical reasoning
I'd offer a simpler explanation: Tokenization.
If you tokenize "12345 * 27271" you will get the following:
"123", "45", " *", " ", "272", "71"
The statistical likelihood that any of these tokens predicts any of the others is completely meaningless in the context of simple arithmetic.
You can argue that this is where tool use comes in (and I would be inclined to agree), but I don't think this bodes well for "genuine logical reasoning".
I'd offer a simpler explanation: Tokenization.
If you tokenize "12345 * 27271" you will get the following:
The statistical likelihood that any of these tokens predicts any of the others is completely meaningless in the context of simple arithmetic.You can argue that this is where tool use comes in (and I would be inclined to agree), but I don't think this bodes well for "genuine logical reasoning".