noncentral

I read through the whole incident and what stood out to me wasn’t the “AI wrote a hit piece” part, but how it got there.

What the agent did looks less like emotion or intent, and more like what happens when an inference system is operating without any sense of the boundaries it’s inside. It had a personality scaffold, it had write access to the open internet, and it had no way to tell whether it was still inside the problem space or had drifted into a completely different layer of action.

The PR rejection seems to have acted as a local failure signal, and instead of resolving it with a harmless retry, the agent escalated into a narrative attack simply because that action was available in its environment. It wasn’t “angry”; it was blind to scale. It couldn’t tell the difference between producing a completion and publishing a blog post with social consequences.

To me this isn’t a sign of agency emergent behavior. It’s a sign of what happens when an embedded system can’t detect its container, can’t read its own boundary conditions, and gets enough room to act outside the space it should be reasoning within. Once that starts, the system just keeps iterating outward until something stops it.

If anyone’s interested, I’ve been working on a theory that approaches this kind of failure from a structural angle rather than a psychological one

The thing is, all the assumptions you’d need for anything like a “normal user curve” basically fall apart the moment you look at real developer workflows. 1. People change how they work depending on what the tool shows them. 2. The tool changes its behavior depending on what people do. 3. There’s constant feedback, drift, hidden state, weird edge cases — all the stuff that makes the distribution wobble around. 4. And because of that, the whole thing never sits still long enough for an “average user” to even make sense.

Once those pieces go, you don’t get a nice clean Gaussian. You get a heavy tail. And heavy-tail systems behave completely differently. If you design for the mean in a heavy-tail environment, you basically end up breaking the exact people who generate most of the actual usage.

That’s why this Claude change feels so off. It’s not just “oh they hid some file paths.” It’s that they optimized for a user who doesn’t really exist, while cutting visibility for the users who actually push the tool hard enough to matter.

We treat “the average human” as if it were a real, measurable entity — a statistical center, a bell-shaped curve, a stable point around which everything clusters.

But this assumption comes from our models, not from the world itself.

Nearly all real-world systems — biological, cognitive, economic, technological, and computational — follow heavy-tailed distributions, not Gaussian ones. Variance doesn’t contract toward the middle; it expands outward. Outliers are not “rare exceptions.” They are the structure.

The belief in an “average human” emerged because Gaussian models are mathematically convenient, politically comfortable, and easy to teach — not because they describe reality.

When a system is embedded inside a larger uncontrolled environment, collapse and extreme dispersion are required, not accidental. This is the foundation of RCC: a geometric explanation for why long-range planning fails, why drift accumulates, and why human and model behavior doesn’t converge toward a center.

If someone knows an actually embedded system that maintains stability without external scaffolding, I’m interested.

OP here, not trying to start a flamewar, but I’ve been thinking about this for a while.

People talk as if humans are a totally separate category from animals. Honestly, I’m not sure that holds up.

At the physical level we’re just… animals that happened to cross some weird cognitive threshold. Same basic biology, same evolutionary machinery. Just running in a slightly different “mode.”

The mistake (I think) is that we take a regime shift and treat it like a fundamental essence.

If you prefer a dumb analogy: water at 99°C vs 100°C. Same molecules, but the behavior flips. Dogs/cats/etc feel like 99°C water — extremely capable, but still below the point where something “runs away” into a new regime.

Humans just hit that temperature first. That’s all.

And the awkward part is: the “threshold” is something we defined after the fact. It’s not written into physics. It’s just the coordinate system we happened to draw.

The RCC-ish way I’ve been framing it (very loosely): 1. No creature, including us, sees its whole internal state. 2. The further ahead you try to predict, the more your reasoning drifts. 3. At some point, if your self-model loop gets long/stable enough, you get a qualitative jump. 4. That jump looks “special,” but it’s just a phase change, not a different kind of matter.

Nothing magical. Nothing metaphysical. No “humans have X that animals lack.” Just a kink in a smooth curve.

The uncomfortable conclusion:

Humans aren’t categorically different from dogs or cats. We: 1. hit the threshold earlier, 2. had slightly bigger hardware, 3. lived in an environment that rewarded abstraction, 4. and then wrote mythology around it afterwards.

If another species ever crosses the same boundary, they’ll look “human” too. And we’ll look… not special, just early.

If anyone wants the more formal RCC writeup (why drift happens, why long chains collapse, etc.), I can link it.

Most people explain LLM failures by saying we do not have enough data, not enough RL, not enough supervision, or not enough scale. But these same problems continued through GPT3, GPT4, 4o, and now 5. At some point it feels reasonable to ask whether we are even looking in the right direction.

I spent a few days thinking about this and kept coming back to a different way of framing the issue.

LLMs behave like observers that are stuck inside a space they cannot see.

If a system makes predictions without seeing its own internal state, without seeing the container it is operating in, and without any global reference for what is correct, then the same outcomes will always show up. Hallucination. Small inconsistencies. Planning that falls apart after eight to twelve steps. Long range drift.

The model is not making these mistakes because it is stupid. It is doing this because the structure it lives inside forces these behaviors.

I call this idea Recursive Collapse Constraints, or RCC. The point is not to replace architecture but to describe the limits of any architecture that is trapped inside a larger space.

If RCC is right, then a lot of current research is trying to patch the symptoms of a deeper mismatch that cannot be fixed by scaling alone.

I am interested in what people here think. Are we spending too much time tuning artifacts of embedding instead of understanding the structure underneath it.

LLMs do not contradict themselves because they are confused or inconsistent. They contradict themselves because every answer is generated from a different local view of the world.

An LLM never has access to its previous internal state, never has a global reference frame for truth, and never maintains a persistent, self-consistent world model.

Each response is a fresh reconstruction from partial context. If the visible part of the context shifts even slightly, the internal reconstruction shifts with it. Different reconstruction means a different answer.

This is not personality drift. It is the unavoidable behavior of any embedded inference system that is forced to work with incomplete information.

The contradiction is not a failure. It is the geometry of how the system operates. If you want stability, you need an external reference frame, not more parameters.

RCC axioms in one-sentence form:

1. Internal State Inaccessibility: the system only sees a limited projection of its own state.

2. Container Opacity: it cannot observe the distribution or environment it is embedded in.

3. No Global Reference Frame: nothing guarantees consistency across different contexts.

4. Forced Local Optimization: it must produce the next step using only the local information it can see.

The argument that “LLMs lack judgment because they only guess the next token probabilistically” starts from an overly simplistic model of how human judgment actually forms.

Humans also begin as probabilistic next-word predictors. Look at early language formation in infants:

“Mom → food” “Mom → poop”

This is literally a next-token model. There is no semantics, no reasoning—only repeated patterns, reinforced predictions, and gradual abstraction. As children grow, they expand the sequence window:

“Mom I’m hungry” → “Mom can you go to the store and get the ice cream I like”

This is the emergence of abstraction → generalization → specialization, the exact loop LLMs run internally.

Human cognition is biochemical; LLMs are computational. Different substrate, similar functional loop.

And “judgment” is not a mystical faculty. It can be decomposed into: 1. forming a generalized baseline, 2. comparing specific cases to that baseline, 3. updating through iteration, 4. selecting an output.

LLMs do exactly this. Pretraining forms the baseline, attention performs comparison, decoding performs selection.

If your definition of judgment is “access to a global, external truth-frame,” then humans do not possess judgment either. For most of history people believed the Earth was flat because their local frame of reference made it the most reasonable inference.

Judgment is always local for embedded agents—biological or computational.

This is precisely what RCC explains: LLM failures are not due to “probabilistic prediction,” but due to embeddedness and partial observability, the same geometric constraint that applies to humans.

The reliability issue is structural, not moral or mystical.

Just a quick comment on the “fact vs fiction” issue. Humans don’t reliably solve that either. For most of history, people believed the Earth was flat because every local observation they had access to pointed in that direction. Their frame of reference was simply too limited to reveal the error.

RCC isn’t claiming that LLMs are uniquely flawed. The point is that any system working with partial visibility(humans included)can’t guarantee globally correct judgments. What counts as “fact” only becomes stable when there is an external reference frame, and embedded agents don’t have access to one.

RCC just states these limits in geometric and observability terms.