enjeyw

The author hints at this but it seems like one issues is that while JEPA is good at distinguishing between unpredictable noise and predictable features, the model has no way of assigning importance to different predictable features.

So for a system where it’s very difficult to exactly reach the desired end state, the model needs to choose between (for example):

- reaching a relatively achievable scene where 95% of the features in the latent are correct, which includes stuff like visible enemies, Mario’s position on the screen etc

- reaching a far more difficult to access scene where there’s a bunch of differences in the actual level visuals, but theres a match on the latent for the tiny set of pixels in HUD that indicate you’ve hit the victory condition

We obviously know that it’s not good enough to reach an early scene that looks similar to the victory condition but isn’t. The model doesn’t.

In a sense, this is what the linear probe helps with - it allows us to re-weight the latent and say “actually, while the latent encodes many things about the world, the thing we really care about is the X position.”

I’d be curious what happened if rather than planning actions on cross entropy of a final scene, the model just tried to find the actions that maximize the predicted X value of the probe.

I think that’s a false dichotomy.

My Lazer Genesis Helmet is a MIPs and it’s the lightest helmet Lazer made at the time.

Much more breathable than my previous helmets too.

Yeah I did wonder myself if that tweet was an admission of guilt.

If I were a lawyer responsible for defending Trump in the Hague, I'd argue that the tweet was actually an abbreviated way of saying "If Iran does not comply, we will destroy all military assets, including but not limited to their ICBMs, Bridges, and Power Stations, such that we have total military dominance."

Now very obviously (to me at least) this was not the intent of the message, but I don't know whether you could prove that in a hypothetical war crimes trial.

What I find tricky to reason about here is that whether destroying infrastructure comes down to "whether the military advantage outweighs the impact to civilians", and as far as I can tell, there's no robust way to assess this.

Indeed, this seems to be what supporters of Trump are leaning on, as you can make the argument that _any_ bridge, or _any_ powerplant could hypothetically be used by the military, and that this conflict is sufficiently important for the livelihood of people in America/"The West" that doing anything that even slightly helps tips the odds is justifiable.

Ok I’ll bite…

37 miles?!? Why??

Oh I didn’t know that! Thanks for the clarification.

Well markets are evaluated on a number of different metrics depending on what you’re trying to determine.

If you want to go be pedantic about it and select one metric, markets are evaluated on their Brier Score or some other Proper Scoring Rule, not accuracy.

However, I prefer calibration as a high level way to explain prediction market performance to people, as it’s more intuitive.

It absolutely has been!

In general prediction markets can’t be “correct” or “incorrect” - for instance if a prediction market says there’s a 60% chance of an event occurring, and it doesn’t occur, was the market right or wrong? Well it’s hard to say - certainly the market said the event was more likely to occur than not, but only just, and who knows? Maybe the event _only just_ occurred, and very nearly didn’t!

So generally we say a prediction market is “correct” if it is “well calibrated”, which is to say that if we took all the events that the market said had a 60% chance of occurring, then approximately 60% percent of these events occurred (with the same holding true for all other percentages).

On this note, an interesting phenomenon that used to occur was “favorite-longshot bias”, where markets would consistently overestimate the likelihood of longshot events occurring - so events that the market predicted would occur 10% of the time would only occur 5% of the time. What’s fascinating is that once people realized that this bias exited, they began to exploit it by making bets against longshots, which had the effect of moving the market and removing the biases, making the markets well calibrated. It’s a pretty neat example of the efficient market hypothesis in action!

I used to share a somewhat similar sentiment.

I know one anecdote is not data, but his investment in BYD all the way back in 2008 does counter that viewpoint somewhat - his investment success in the BYD case isn’t from other investors following him in, it’s from him identifying BYD as a successful company far before any other major investors did.

Overly specific LLM research into KV cache eviction.

The vast majority of tokens in a sequence will be irrelevant to an attention mechanism outside of a very small window. Right now however we tend to either keep all cache values forever, or dump them all once they hit a certain age.

My theory is that you can train model to look at the key vectors and from that information alone work out how long to keep a the token in the cache for. Results so far look promising and it’s easy to add after the fact without retraining the core model itself.