Comment by danlitt

Not sure about multiplication, but letter counting is still constantly wrong if you can trick the LLM into not calling other tools. Obviously there are lots of mitigations on the server side to try to avoid this happening, but the underlying LLM has not got any better at this type of problem (and probably can't).