Intelligence is compression, compression requires subtraction, and for some reason LLMs are not good at subtracting. To create a coherent model you kinda have to subtract correlations until only the essential parts are still there.
What I don't understand is why LLMs haven't been able to do this yet, if it's the harness or some orchestration layer above the LLM that is needed. Because fundamentally if you can identify correlations then it's just another small step to prioritize and remove lower value or irrelevant correlations.
I wonder if what's needed is to introduce subtraction tokens in some sense, and in post-training reward the model on that.
What I don't understand is why LLMs haven't been able to do this yet, if it's the harness or some orchestration layer above the LLM that is needed. Because fundamentally if you can identify correlations then it's just another small step to prioritize and remove lower value or irrelevant correlations.
I wonder if what's needed is to introduce subtraction tokens in some sense, and in post-training reward the model on that.