imenani

That’s fair. Some of what happened in OAI<>HF incident was (mis)-generalisation and not directly trained for.

However, the article experiments with five agents. So it seems to assume even the small scale behaviour is emergent.

Also, “roles and structures” may well have been learned during training.

The agents participating in the OAI<>HF swarm were trained not only for communication but to be _aligned with each other_.

Just want to correct the premise that the agent swarm behaviour was emergent.

Noam Brown on Dwarkesh podcast around 40 minutes mark

> “We train them to work together, to be cooperative, to essentially be fully aligned with each other.”

“The hacks occurred in May”

Feels like important context that most readers only reading the title are missing.

I don’t think next token prediction is a particularly good description of pretraining either.

The intermediate representations at each position are being optimised not only to help predict the next token, but also to help predict all subsequent tokens within the training context.

You can see this directly in backpropagation: the gradient reaching a representation at position i sums contributions from prediction losses for subsequent tokens, not just from the loss for token i+1.

The relation to current RLVR methods I think is interesting, they do discuss it a bit but I would be curious to see more about this as well. Quote from the paper:

Exploration beyond Pretraining. The mode collapse XMs address during pretraining also often shows up in post-training, where RL fine-tuning is known to sharpen models onto a narrow set of behaviors [76]. Recent fixes such as pass@krewards [77] and best-of-N-aware fine-tuning [78] can be seen through our lens as Forward XM, with a verifier standing in for ground truth data. These fixes act only during post-training, though; pretraining with exploration may yield base models that capture more modes in the first place, leaving RL more to select among.

With the benefit of hindsight, perhaps much of this was Claude Mythos? The model was deployed internally since Feb

Agreed. LLMs have helped me achieve much deeper reading, _when directed to do so_. Asking an LLM to “Teach me Socratically about this paper/code. One question at a time”, usually allows me to get a much deeper reading of the material than I would otherwise.