LinchZhang

HF was OpenAI's agents not Claude.

As I said multiple times in the linked post, everybody knows that CoT monitoring is imperfect.

The question is whether the benefits are high enough that you should be happy to drop an imperfect safety mechanism in the hopes that some day, one day, you'd get a better one. Seems like the wrong tradeoff to me but regardless, arguing that "CoT monitoring is imperfect therefore we should drop it" when we do not in fact have a better mechanism in place is silly.

This is not how we do things in any other engineering discipline or risk-mitigation system.

It's like saying we shouldn't have rapid antigen tests because they have nonzero false negative rates, or not writing software tests because the tests never catch all bugs.

> Not everyone in the AI space approves of these people or their doomerish.

"Not everyone in the automobiles space approve of consumer safety advocates" What an understatement!

It's very hard in practice. Even without optimization against it, it's about as hard as understanding an activation layer today and will probably get harder in the future.

There's also nothing stopping the secondary LLM from confabulating bullshit and there are less checks on it than CoT (harder for either humans or other models to externally verify).

I agree in the long run whitebox interpretability would be better than CoT monitoring but the technology is very much not ready for it today (and it's not clear we'd solve enough of interpretability before the AIs have actually scary capabilities).

I agree CoT monitoring is imperfect but it's okay in practice and helps us get defense-in-depth re: model intent. We absolutely do not have a singular safety mechanism in place that's sufficiently good that we can use it in exclusion of all other imperfect ones.

"Do you know what is a faithful representation of what the model wants to do? Tool calls."

tool calls could absolutely be spoofed, my impression is that this happened many times in the OAI HuggingFace attack.

"Phoebus keeps taking screenshots of our latest model’s thoughts. It’s getting kind of embarrassing."

Thank you for your kind words and empathy! I appreciate it. Writing to the void is hard, and while I care a lot about improving and not being wrong, I also appreciate it when people realize that the writer on the other side of the screen is a real person, and deliver their feedback with kindness and empathy.

Thanks, appreciate the anecdote!

If you have suggestions on a better title, please let me know! I tried pretty hard to come up with different ones, giving time constraints, and this was the best one I had. I'm really bad at titles and this is an area of active growth for me :)

Why fixate on a specific word rather than the overall idea? If the idea is clear enough I don't think it's worth fighting over semantics.

I agree I used the word substantially more expansively than some other people use it. That's why I defined it in the beginning so people can understand the local scoping of the relevant word! :)

(That said "salespeople" are in the middle layer under your definition as well)

The other term I was thinking of using for this post was "bullshit jobs." So titling my post "bullshit jobs are real jobs" but I didn't want to fight against the motte-and-bailey of specific jobs being possibly bullshit jobs.

("coordinators" presumed the conclusion too much and also points to a specific thing )

I think many people have some intuition that work can be separated between “real work“ (farming, say, or building trains) and “middlemen” (e.g. accounting, salespeople, lawyers, bureaucrats, DEI strategists). “Bullshit jobs” by David Graeber is a more intellectualized framing of the same intuition. Many people believe that middlemen are entirely useless, and we can get rid of (almost) all middleman jobs, RETVRN to people doing real work, and society would be much better off.

Like many populist intuitions, this intuition is completely backwards. Middlemen are extremely important! Coordination problems are real problems, and the bottlenecks to global wealth and flourishing.

The post goes into details for why.