mynti

Again, very impressive from a technical point. But language is so flawed to do anything creative. It just turns out all looking the same and kind of boring

> If you consider how impossible the startup journey is, it stands to reason that only through intensity can something be created out of thin air that grows to be worth billions of dollars in just a few years.

It is sad to me that this is the benchmark for so many people. It is only a success if it is worth billions.

Can someone explain this architecture a bit more in depth? They say the pointer head scores the hidden state at each option against the hidden state of the answer. But the LLM produces hidden states per token, so an option can span multiple tokens, no?

The tech has always been mature enough but simply too expensive. But since combustion engines are sold out for the next years, data centers are rushing to get anything they can. So the extra millions for power generation will be a rounding error

It does not. But there are so many sources that can be. And easily at that. So why go this route?

This feels very useful for training agents but almost completely useless to humans? Do I really want to go back to every change I made? Whatuse would that be?

To me it feels so weird that people are trying to push these model for online shopping like "here is how this dress/shirt/pants would look on you". But these models will always make the clothes fit your body and show you in flattering light and so on. How the actual garment fits is still as elusive as before these tools

The "climate" cares little about carbon intensity of labor or dollars. It cares about absolute tons of green house gases in the atmostphere and if you use more to produce more you still use more.

Very cool! How hard would it be to add all the solar installations? They must be in the register as well

What prevents a bad actor from posting "easy" problems to the board, solving them and getting quick reputation. Then as a validator they can easily validate their own malicious changes to someones software?

I always wondered how these models would reason correctly. I suppose they are diffusing fixed blocks of text for every step and after the first block comes the next and so on (that is how it looks in the chat interface anyways). But what happens if at the end of the first block it would need information about reasoning at the beginning of the first block? Autoregressive Models can use these tokens to refine the reasoning but I guess that Diffusion Models can only adjust their path after every block? Is there a way maybe to have dynamic block length?

Does anyone know what kind of RL environments they are talking about? They mention they used 15k environments. I can think of a couple hundred maybe that make sense to me, but what is filling that large number?

With the rough numbers from the blog post at ~1k tokens a second in Cerebras it should put it right at the same size as GLM 4.7, which also is available at 1k tokens a second. And they say that it is a smaller model than the normal Codex model

I think this says a lot about the business approach of Anthopic compared to OpenAI. Just the vast amount of free messages you get from OpenAI is crazy that turning a profit with that seems impossible. Anthropic is growing more slowly but it seems like they are not running a crazy deficit. They do not need to put ads or porn in their chatbot

I feel like this comes from the rigorous Reinforcement Learning these models go through now. The token distribution is becoming so narrow, so the models give better answers more often that is stuffles their creativity and ability to break out of the harness. To me, every creative prompt I give them turns into kind of the same mush as output. It is rarely interesting