camuel
Joined 14 years ago
Joined 14 years ago
Some are role playing forensic investigators here. The guy admitted that he got overexcited and submitted unfinished work which he doesn't even remembers which build it was exactly and he published his current source code as OSS which scores even better and fixes all the accusations.
The bitter lesson is that simply scaling training on more games—including self-play—trumps any hand-crafted human input, whether that's fine-tuning on human commentary or clever engineering tricks.
Current models are just high-dimensional interpolation engines. The denser the data sampling, the more accurate the interpolation gets. Given a choice between denser sampling and anything else, denser sampling always wins. That is the bitter lesson.
Computer chess is the canonical example of this.
Sorting-networks which were already mentioned seems similar but a bit too abstract.
My code above doesn't contains values but those are easy to add I think. Of course it is better to permute fixed size pointers / offsets and not the entire blobs which can be of variable size and then it will complicate everything beyond feasible for SIMD
But I'm not sure, it's my interpretation. The write-up resonates with me but I can see it as a bit ambiguous and inviting many incompatible interpretations.
Proper treatment of the subject in a book format would be great.
However, Intel dominance is not to be underestimated, they definitely can make industry wide impact quickly. Just saying you can not easily draw parallels here