andblac

Nicely done. For me the most fascinating thing about attention heads is the place where Attention matrix is already computed and is getting multiplied by Value vector. It behaves exactly like pushing Value vector through Dense layer of ordinary network where Attention matrix forms weights of that layer. So attention head is trained to construct this small single layer network dynamically during inference from Key and Query. And that's the point. That's rarely underlined in explanations of LLMs architecture and for me it's quite amazing that it works so well. This mechanism easy to observe in this particular visualization if you click through it.

The "ALL CAPS" part of your comment got me thinking. I imagine most llms understand subtle meanings of upper case text use depending on context. But, as I understand it, ALL CAPS text will tokenize differently than lower case text. Is that right? In that case, won't the upper case be harder to understand and follow for most models since it's less common in datasets?

And now we’ve built LLMs - the biggest mirror of them all. The Internet and smartphones gave us countless ways to look at ourselves and see how others see us. And then LLMs helped us gaze at the sum of all that and even confront reflections of our own thoughts.

Yeah, TFA ended just before it got to the really interesting part of how self-reflection itself is fundamental to the development of concisousness. Mirror-like technologies don't just show us our own appearance. They help us understand how we relate to the world around us.

It reminds me of Kieślowski's movie Camera Buff (1979), where the main character in iconic scene points the camera at himself and realizes that the act of making movies reflects not only his subjects, but also on who he is in relation them.

Yeah, I'd would love to read article on all that.

At first glance, this reminds me of how branch prediction is utilized in CPUs to speedup execution. As I understand it, this development is like a form of soft branch prediction over language trajectories: a small model predicts what the main model will do, takes few steps ahead and then verifies the results (and this can be done in parallel). If it checks out, you just jump forward, it not you take miss but its rare. I find it funny how small-big ideas like this come up in different context again and again in history of our technological development. Of course ideas as always are cheap. The hard part is how to actually use them and cash in on them.

Skimming through the source it seems to run 'car' and 'person' objects through llava with the following prompt:

- "person": "get gender and age of this person in 5 words or less",

- "car": "get body type and color of this car in 5 words or less".

So YOLO gives the bounding box and rough category, while llava describes the object in more details.