Comment Comment by sithamet > The best performing model scores a 66.8. So does this prove LLMs can “understand” the code? I would argue that the answer to this is “no”.By this benchmark, the human who makes mistake in an intendedly obscure spec does not understand the code either.But you did show these pesky stochastic parrots their place!But they can simply decompile the language, should it desire, so we need to think deeperMy next post will be "I built a programming language that LLMs can't use (REALLY)", but it's in a private repo so clunkers can't get to it parent sithamet 19 days ago
By this benchmark, the human who makes mistake in an intendedly obscure spec does not understand the code either.
But you did show these pesky stochastic parrots their place!
But they can simply decompile the language, should it desire, so we need to think deeper
My next post will be "I built a programming language that LLMs can't use (REALLY)", but it's in a private repo so clunkers can't get to it