Ignoring its negative connotation, it's more likely to be a highly advanced "stochastic parrot".
> "You don't do that without some kind of working internal model of mathematics."
This is speculation at best. Models are black boxes, even to those who make them. We can't discern a "meaningful internal representation" in a model, anymore than a human brain.
> "There is just no way you can get to the right answer by spouting out plausible-sounding sentence completions without understanding what they mean."
You've just anthropomorphised a stochastic machine, and this behaviour is far more concerning, because it implies we're special, and we're not. We're just highly advanced "stochastic parrots" with a game loop.
> "You don't do that without some kind of working internal model of mathematics."
This is speculation at best. Models are black boxes, even to those who make them. We can't discern a "meaningful internal representation" in a model, anymore than a human brain.
> "There is just no way you can get to the right answer by spouting out plausible-sounding sentence completions without understanding what they mean."
You've just anthropomorphised a stochastic machine, and this behaviour is far more concerning, because it implies we're special, and we're not. We're just highly advanced "stochastic parrots" with a game loop.