addag

I haven't read this paper in details, but I have read papers about the platonic hypothesis in the past. Usually one of their weakness is about the design an experiment proving that "neural networks converges towards the same latent space when scaling". Usually authors are effectively proving a subset of the hypothesis, that can be explained by the size of the model and the transfer learning part, leaving not definite answer.

Here it seems that their definition is more robust (the representation is defined up to a rotation), but I am wondering how much more re-training they need to perform.

> So if you find ways of improving training efficiency (the simplest conceptually is probably local/global sliding window attention like Gemma; the latest DeepSeek and Kimis have quite fancy ones), you will be able to have more effective compute, and train smarter models while keeping data and param count the same.

I don't get how changing the attention head (i.e sliding window attention instead of vanilla transformer) is only seen as a "compute" improvement. Indeed, it improves the complexity from quadratic to linear, but there was a reason why people used transformer and not sliding windows for so long (they were already there in 2021).

Making sliding window attention work with performances on par with vanilla transformers is a non trivial problem and is not solved by only putting more compute. This still requires "smart" ideas in that direction.

Indeed it is not faithful, but it is ones of the less bad ones we have for now. And removing it definitely goes towards the wrong direction in terms of intepretability.

Very interesting read and nice animation. One thing though is that the solution seems "too good to be true". Lots of smart people have worked on intepretability problems before and tried the simple ideas he is talking about.

If those results hold, it is a real breakthrough in the sense that it is a sharp simplification of an actually complicated problem.

Well technically my take is still valid

> Write me a short story

>

> Short story

This looks more like a dating app simulation than a chatbot

Our only hope is that work will cease to exist.

I repeat, I hope you are right as I enjoy coding a lot. I just wish that people who keep not using LLMs too much won't have troubles because of that.

Putting aside the bond I have towards with the FOSS community, I am curious of what will be the new equilibrium of this ecosystem.

As much as I'd love to believe it, it is now a conservative take. Sure, having a solid architecture in mind still matters right now, but manually writing code is completely unnecessary and, before long, even designing the architecture is going to be completely automated.