ImageXav
Joined 5 years ago
Joined 5 years ago
Paper: https://arxiv.org/abs/2401.00036 Project page: https://discrete-distribution-networks.github.io/
Given that these were published at ICML 2025, at which the author was a top reviewer as per their own website https://alexiglad.github.io/, I wonder how influenced they were to pursue this avenue of the back of it. They do have a related works section in appendix E, but it somehow seems to miss this. Which is odd, as the paper made a splash at the time at the conference and even made it close to the top of hacker news due to the novelty.
Of course, DDNs are a fundamentally new architecture, whereas this is more a generaliseable training strategy, but there is a very similar core.
Kim, however, has exposed the whole reasoning trace, or enough of it to matter. I'd almost forgotten how nice it is to see this. I've been able to see all of the weird twist and turns it takes and it is joyful. But also, far, far more informative and means I can debug ideas far more thoroughly. Also, at a first glance it seems to have gotten quite far on a niche hobby horse of mine that no LLM has been able to crack. I'll be testing this more for sure.
The structure of the code is easily readable as I enforce concenventions followed by good libraries. And I can easily plug in new datasets. It's pretty good frankly.
I did feel with the change in editorial direction a while back that they lost some of their edge. I've since mostly just stuck to the Financial Times. It feels less worldly, but the content of the articles feels better.
[0] https://acsjournals.onlinelibrary.wiley.com/doi/10.3322/caac...
I do think a lot of folk would benefit from the introspection this offers. We've all been given the opportunity to become middle (and middling) managers, and a lot of the challenges we face are those of people who direct. Setting direction is tough. But LLMs are awesome tools.