MauranKilom

But you posted this to a site that is literally called Hacker News... To be clear, I am not supporting any attempt at undermining your project, but people are pointing out to you that your results will be called into question if your only defence against "hacking" is "I hope people don't figure out how to do that".

Because no message has ever had its meaning twisted by overzealous spellcheck. No ducking way.

I don't have much to contribute, but the name "Rephsis" seems borderline unpronounceable to me (maybe that's why the empire is crumbling? :P). I don't know whether you had some inspiration behind it, but to me it has a strong "I combined letters until I had a word that looked new" vibe.

The screenshots in the Steam page do look impressive though! As an outsider, I could mistake this for a recent Civ game. Congrats on making it here!

Do they? I think it's mainly about how much energy you are willing to invest into making rational decisions over emotional/"gut" ones. I mean, in the end, it probably doesn't matter which kind of shaving cream you buy, so it's easiest to go with the one that (due to e.g. ad-created familiarity bias, or just "niceness" of visual presentation) you have the best feeling about.

In other words, you introduce an invalid state to every object and make constructing objects a lot more cumbersome. The first is the exact opposite of the (imo highly desirable) "make invalid states unrepresentable" principle, and the second is also a pretty extreme cost to productivity. I wouldn't say this is never worth it, but it's a very high price to pay.

Thanks for this, it provides an intriguing approach to thinking about transformers (or predictors in general)!

For extracting the fractal from the residual stream, did I understand it correctly as follows: You repeatedly sample the transformer, each time recording the actual internal state of the HMM and the (higher-dimensional) residual stream. Then you perform a linear regression to obtain a projection matrix from residual stream vector to HMM state vector.

If so, then doesn't that risk "finding" something that isn't necessarily there? While I think/agree that the structure of the mixed state representation is obviously represented in the transformer in this case, in general I don't think that, strictly speaking, finding a particular kind of structure when projecting transformer "state" into known world "state" is proof that the transformer models the world states and its beliefs about the world states in that same way. Think "correlation is not causation". Maybe this is splitting hairs (because, in effect, what does it matter how exactly the transformer "works" when we can "see" the expected mixed state structure inside it), but I am slightly concerned that we introduce our knowledge of the world through the linear regression.

Like, consider a world with two indistinguishable states (among others), and a predictor that (noisily) models those two with just one equivalent state. Wouldn't the linear regression/projection of predictor states into world states risk "discovering" the two world states in the predictor, which don't actually exist there in isolation at all?

Again, I'm not doubting the conclusions/explanation of how, in the article, that transformer models that world. I am only hypothesizing that, for more complex examples with more "messy" worlds, looking for the best projection into the known world states is dangerous: It presupposes that the world states form a true subspace of the residual stream states (or equivalent).

Would be happy to be convinced that there is something deeper that I'm missing here. :)

FWIW, "number of flights you take" also drowns out your eating habits in environmental impact. Compared to how much they cost, flights have stupid CO2 equivalents.

However, I don't know why you are comparing a single billionaire vs a single X kid household. Like, the number of each (or even of private jets) are not even _remotely_ in the same ballpark. Which is why "number of kids" is not at all a strange place to focus on environmental impact, but "billionaire lifestyle choices" is.

Don't you think that we would be able to observe some anisotropy in the data if that were the case? (Note: I have no clue, but it seems unlikely...)

As a notable example, the MP3 format was hand-tuned to vocals based on "Tom's Diner" (i.e. a female voice). It has been accused of being biased towards female vocals as a result.