x1000

It’s the negative of the inverse of the golden ratio. (Also 1 minus the golden ratio.) So, good for anything the golden ratio itself is good for.

Not a physicist, but I see it this way too. My understanding of Boltzmann brains is that they are a theoretical consequence of infinite time and space in a universe with random quantum fluctuations. And that those random fluctuations would still be present in an otherwise empty universe. So then this article has no bearing on the Boltzmann brain thought experiment or its ramifications.

If they had experimented using a newer model (gemma 3, deepseek-1 7b, etc.) and reported better results, would that be because their newer baseline model was better than the llama 2 model used in the previous methods' experiments? A more comprehensive study would include results for as many baseline models as possible. But there are likely other researchers in the lab all waiting to use those expensive GPUs for their experiments as well.

I ran into exactly same pain point which was enough to nullify the benefits of using zod at all.

Could you help explain how we would achieve an attention score of exactly 0, in practice? Here’s my take:

If we’re subtracting one attention matrix from another, we’d end up with attention scores between -1 and 1, with a probability of effectively 0 for any single entry to exactly equal 0.

What’s more, the learnable parameter \lambda allows for negative values. This would allow the model to learn to actually add the attention scores, making a score of exactly 0 impossible.

Imagine you are a LLM and all you see are tokens. Your job is not only to predict the next token in a sequence, but also to create a nice embedding for the token (where two similar words sit next to each other). Given a small enough latent space, you're probably not concerning yourself too much with the "structure inside" the tokens. But given a large enough latent space, and a large enough training corpus, you will encounter certain tokens frequently enough that you will begin to see a pattern. At some point during training, you are fed:

1) An English dictionary as input.

2) List of words that start with "app" wiki page as input.

3) Other alphabetically sorted pieces of text.

4) Elementary school homeworks for spelling.

5) Papers on glyphs, diphthongs, and other phonetic concepts.

You begin to recognize that the tokens in these lists appear near each other in this strange context. You hardly ever see token 11346 ("apple") and token 99015 ("appli") this close to each other before. But you see it frequently enough that you decide to nudge these two tokens' embeddings closer to one another.

Your ability to predict the next token in a sequence has improved. You have no idea why these two tokens are close every ten millionth training example. Your word embeddings start to encode spelling information. Your word embeddings start to encode handwriting information. Your word embeddings start to encode phonic information. You've never seen or heard the actual word, "apple". But, after enough training, your embeddings contain enough information so that if you're asked, ["How do", "you", "spell", "apple"], you are confident as you proclaim ["a", "p", "p", "l", "e", "."] as the obvious answer.

An interesting aspect of these cryptocurrencies is the aspect of consensus, not through the intended mechanisms like PoW, but through societal acceptance. Look at BTC and BTG (bitcoin and bitcoin gold). One has the suffix of "gold" while the other maintains the (arguably?) superior lack of any such embellishments/augmentations. Was it the miner's decision to call it that? Was it the users?

Look at Ethereum v Ethereum classic. Same deal. We have two chains that share a common history, yet at some point the users of both decided to split and then society had to come to a consensus on what each chain would be called. Again, did the miners sit around and conspire to which chain would be called "Ethereum?" I don't think so. I think the decision was decentralized and emergent.

My point is, even if there was a nefarious actor who attempted a 51% attack, it seems like there would be enough of a societal pressure to ignore their empty blocks. There would exist a chain that would still be valued by the perpetrators, but not so much by the individuals being harmed by such an attack. The attacked chain would be maintained and acquire a new name "Bitcoin Hacked" or something similar, and the chain where society ignores the empty blocks would go on its merry way still being called "bitcoin."

Been following this guy for a while now too. And you're absolutely right. It's been fun watching his evolution. Glad to see this series of articles posted here. I've been sharing this series with my coworkers and we've since gotten into Haskell, Elm, and just taking more functional approaches in general in our primary stack.

Is there a reason you specified partial derivative with respect to time? Why not just d/dt(state)?

I've been tempted lately to model application state as the integral over time of all events (deltas) that have occurred. For example, imagine a simple game state:

  type GameState = {
    x: float; // x coordinate
    t: float; // current time
    v: float; // velocity variable
  }

If initial game state =

  { x: 0,
    t: 0,
    v: 0, };

and you have some events (deltas):

  [{t: 1, v:1}, {t:2, v:-1 /*this is deltaV, so back to v: 0*/}, {t: 3, v:2}, {t:5, v:-2}]

You could "sum" them up (integrate over time) and get a game state of x = 5, v = 0 where t >= 5.

I've found this approach kinda sucks when you add things like collision detection. Your application would have to emit velocity deltas when objects would collide. If you've got a point that bounces around in a box with perfectly elastic collisions, then over time you'd have an infinite number of these collision/velocity delta events. But as you receive new events, all your precomputed collision events are for nothing (if your player logs out). So the traditional update loop simulate each tick as it occurs seems to work best.

Is there a more general way of thinking about this integration? Maybe with respect to another variable? Perhaps it would address this problem I'm having where I feel forced to quantize my game state onto ticks.