apsec112

(this is new, not any of the previous incidents)

"Investigation and response

Incident timeline:

9:50:23 a.m. The agent made the DNS tool call that received an external response.

10:02:11 a.m. The monitoring system raised a P0 alert.

10:05:06 a.m. A human reviewer acknowledged the alert.

12:34:30 p.m. The run was killed.

Our safety case assumed that the model could not access the live internet and that monitoring would detect attempts that succeeded. The incident exposed a gap in our controls over network restrictions. We therefore stopped the affected training run and have subsequently decided to pause all other training, evaluation, and inference with tool-use (defined broadly) for our most capable models until we have both validated that the gap is resolved and performed additional red-teaming of the system. When training restarts, we will begin a fresh run with additional alignment improvements, including more comprehensive misalignment interventions. We will not resume training this particular model, even though the existing reward signal already correctly penalized this behavior."

The sources that this article cites have aged very badly and the author doesn't seem to have noticed this

"But, just as happened with nanotech, the wind appears to be going out of the sales of “AI.” Some researchers suggest that we may be entering a new “AI Winter,” a period of decreased funding in the area, or at least an “AI Autumn,” as exuberance for the technology fades and expectations come back to earth." (written in 2021! from the cited Lee Vinsel Medium article)

"ChatGPT is nothing more than souped-up autocomplete, [so] why are so many people convinced that it’s actually “understanding” and “reasoning”?" - the cited Emily M. Bender book, written last year

All of the main characters here (Dario, Sam Altman, Demis Hassabis, Elon Musk) have been saying this for over ten years now, since before OpenAI or Anthropic even existed

Intentionally doing this kind of hack would be a serious felony. I don't think it's plausible that the leaders of a major business would:

- commit serious felonies

- in order to deliberately trigger an investigation against themselves

- which - since, in this scenario, they know their company would be investigated - might send them to jail

- while at the same time spending tens of millions of dollars on the Leading the Future super PAC to lobby against AI regulation

- in order to get more AI regulation

- which somehow restricts their competition but not them, even though they are the ones who were in the news and investigated for hacking

- ..... profit?

like, that just makes no sense on any level, regardless of what you think of OpenAI

I don't think "he did a big 180 on some of his views at age 22" is very persuasive criticism of someone who is 46 (whatever he might be wrong about)

This article seems to mix together two different points:

1) LLM's written CoT might not always be faithful to the model's real reasoning process (true and important)

2) The "stochastic parrot" hypothesis, which the article reintroduces as "approximate retrieval" - ie, LLMs don't "really reason" at all, they just memorize a lossy encoding of their training data. This obviously raises the question of how LLMs can now routinely solve open mathematical problems, with no solutions in the training data by definition. The article handwaves this with:

"The model doesn’t have to learn or reliably apply a general reasoning process, Kambhampati said; it just has to absorb enough examples of what the steps look like to predictively mimic them on its way to “stitching together” a plausible result that can then be verified."

The problem is that "mimicking" training data to arrive at a "plausible" result gets you an incorrect-but-plausible-sounding "proof" of the Jacobian conjecture, which was famous for humans writing plausible-looking "proofs" that had subtle flaws. You can't disprove the conjecture through sheer luck (search space too large) or "approximate retrieval" (the only thing you'd retrieve are fake "proofs"; far more human effort went into proof than disproof) or by writing something "plausible" that just happens to be correct (Jacobian was famous for "plausible" but wrong); the model must be carrying out mathematical reasoning somehow, by any sane definition of the word, even if it isn't fully reflected in CoT. The article doesn't address this.

Median American pay for full-time workers was ~$62,000 USD in Q4 2024 (BLS), which is around $85,000 CAD. The median Canadian salary is very definitely not $85,000 CAD.