Fantastic framework that I will be adopting, thank you!
One thing I will add: I'm sure the optimistic side will try to take the quantization that your (and other work like SlopCodeBench) are introducing and claim that we'll be able to fix the problem by training on that data, producing coding models that are rewarded for coherence.
I invite them to try! But my suspicion is that this will require a rather deep form of intelligence (world models) that the current language/data-based paradigm isn't really equipped to reach without requiring more energy than the Sun.
One thing I will add: I'm sure the optimistic side will try to take the quantization that your (and other work like SlopCodeBench) are introducing and claim that we'll be able to fix the problem by training on that data, producing coding models that are rewarded for coherence.
I invite them to try! But my suspicion is that this will require a rather deep form of intelligence (world models) that the current language/data-based paradigm isn't really equipped to reach without requiring more energy than the Sun.