Comment by goonersallofyou

LLMs are nondeterministic, good luck trying to measure performance at all, much less over time.

Lets say it has gotten worse? What are you going to do about it? Jump to Codex? Then what happen if you perceive that to be getting worse? Jump back to Claude? One of the many problems with these tools.