robrenaud

I was surprised in reading the article that it had nothing to do with the contents of the model. It could have been any model capable of 1M context that triggered this bug/misconfig on Cloudflare/SGLangs side. The title is extremely misleading.

Are there inference products focused on say, optimizing frequently used internal contexts, so that they are compact in the kv cache? Perhaps there rep could be optimized over hundreds+ of distinct real use cases?

Maybe your law firm doesn't want your data going outside your business. But you obviously want high quality analysis.

What is inefficient is thinking in 5000 floats, and then quantizing all that computation into a 17 bit token, just to immediately turn the token back into a vector.

Brains aside, it S1/S2 seems like a good description?

System 1 is quick and intuitive. System 2 is slow and deliberate.

Compared to models that compose responses with many forward passes, thinking in token space, these models make a quick decision after processing their inputs.

Here is a law I wish we had for social media. Let's just align incentives.

If you are working on a social media platform, and your total comp is >= 500k, your children must use your platform for the average amount of time that children of that age range use your platform nationwide.

In the Jev use case, LLMs are horribly uncalibrated. In general, they will not produce good probability estimates.

Their generality also comes with a latency/computation costs.

> That's an interesting choice. One question I had when looking at the jev copy on their blog is if one "line" in their output looks / attends to other lines. I think not, since they say it's parallel and not autoregressive.

I don't understand the connection between the lack of autoregression and options attending to each other.

Non autoregressive models can attend to all the inputs simultanously.

An autogregressive model can can attend to all the options in the context of each other by simply writing the options out twice. Autoregressive models actually requires this, since one of them will come later, and the earlier prefill inputs can't attend to the later ones.

> “Predicting sales conversion probability from conversations using pure Reinforcement Learning”. Can you understand what that means?

I can understand it, and it wouldn't excite me at all.

Jev has a beautiful API and is advertised as something much more general.

> Using briefer abstractions in place of nested loops is almost always preferable.

Indeed, this is why everyone knows the J programming language.

The part where he chooses his inputs to hit worst case behavior in Python's hash function.

Experts vary per token in MoE, there is maximum flexibility. Good for driving down loss, bad for locality/gpu memory/bandwidth.

If expert selection were more constrained, inference systems could take advantage of it. Keeping experts cached would mean not needing to load them from disk/ram every token.