aarondong

The RL dashboard is quite cool.

I wonder if this waters down the “distillation attack” claims by Anthropic. They have their own RL environments! I guess the caveat is that the RL datasets are still opaque, nothing is really proved.

The central thesis is coherent but not particularly substantive. It echoes a lot of existing sentiments in the public about US frontier lab priorities and messaging. “AI creates slop that lowers the quality of media. AI companies have misaligned interests with humanity. AI will try to take your job. Be human, don’t use AI, enrich your life.” Things which are easy to agree on.

There’s a more nuanced discussion to be had about AI when it comes to potential and existing benefits, open source, the geopolitical dynamics, self-hosting, the financial dynamics, the sanctity of human thought, a future of even more personalised advertising and so on. We do need to elevate the discourse around AI and shift the public narratives around what US frontier labs are pushing, understanding their biases but also acknowledging substantive claims, potential benefits and risks.

If Dario is using an unreleased internal model to write, then Pangram wouldn't be able to flag it I suspect. Pangram needs enough public info about the patterns in generated text.

I agree, I like running claude code in my container with auto mode enabled and web access (obv to api.anthropic, even npm for pulling), I will admit. And I can't imagine going back to manually approving each prompt.

I think it boils down to a reasonable expectation of model and harness behaviour. When I use claude code I expect certain guardrails for the model. For these cyber attacks, these models are specifically run without guardrails, on a cyber task, on a lax harness!

I don't think we should force end users to have to worry about agent security, I like long-running agents, but we need to direct regulations towards these actors that know better, have access to base models, and have much more compute than the average person.

Let's continue with the analogy, so in the event of a Waymo running over a pedestrian - who is held responsible?

Probably not the end user, who ordered the Waymo and couldn't reasonably foresee it running somebody over, with the expectation that that the Waymo would legally reach its destination. If the end user tampered with it, they should be held responsible.

An OpenAI team giving an unblocked model access to a lax harness, with instructions to find and exploit cyber bugs in a game exercise, there is probably a reasonable expectation that they can foresee the consequences. With consumer guardrails, it would not have happened.

This isn't about putting constraints on consumers and typical end users, which already have safety filters and use the product with the knowledge that it won't root their machine or start a botnot, but keeping dangerous test runs and other actors experimenting with unsafe harnesses accountable.

But it is an interesting question. When I, a typical user, use a harness and I give an innocuous prompt to my agent in its container, like making a certain refactor, and it somehow escapes and then begins a mass bot attack, there should we more grace given. As agents become more stateful and long-lived, it gets muddy.

This is a business solution, because it means there are legal costs for not having adequate observability and monitoring mechanisms. Every tool call is interfacing with a harness.

But overreach of policy and overregulation would be stifling, so there has to be a threshold to the type of incident investigated, civil or criminal.

Wouldn't the simplest solution for stopping the proliferation of wanton felony generators just be holding operators liable for actions that their agents take? Then the issue is whether the liability is with the model provider or the end user.

If you give an unfiltered agent an open-ended task and equip it with an environment that allows it to execute arbitrary code, a human needs to be held responsible.

My understanding is that the HuggingFace incident would not have occurred with a model that was not an unfiltered internal preview instructed to roleplay an attacker, with access to abundant compute, resources and a slack sandbox to reach its goal.

Embedded human auditors will improve safety standards, but the structural solution is mandating accountability for actual agent operators.

This made me do a triple take. Here is the proposed logic as I follow it:

> AI models are dangerous. They can help bad people do dangerous things. They may be capable of autonomously executing dangerous things. They may cause unwanted effects on the labour market.

> More capable AI models are more dangerous, but require more money to train.

> Money requires investors with expectations that the model will generate a profit over its operational lifetime.

> The operational lifetime value of a model is decreased if every model is public and can be hosted on any infrastructure.

> If investors see less operational lifetime value from model companies, model companies receive less capital and therefore train more capable models at a slower rate.

Follow-ons: There are immediate risks in releasing capable cyber models that can be ablated and then launch cyberattacks. Concentration of power moves immediately into the infrastructure layer for inference.

Models will be kept private for longer, if not indefinitely, given there is less incentive to release them publicly.

Enforcement globally will occur because accessing the lucrative US market means using an open source model.

The entire argument hinges on strict enforcement of this policy, when AI model routing can already be opaque.

It also hinges on investors being rational and expecting free cash flow from AI companies, rather than reaching a criticality threshold of model capability for recursive self-improvement internally and parlaying that into a global mega-corporation.

New labs and companies without infrastructure connections will no longer be able to raise money, given investor expectations, and therefore not be able to increment AI progress.

So a win for the infrastructure layer, models being kept private for longer, new labs being unable to compete, immediate risks in the rollout (I suppose could be mitigated by a staged rollout i.e. policy active in 2030, pricing effects now), enforcement being tricky, and in the event that foreign competitors lead the AI frontier and then start closing models, just conceding the market to them if US consumers and companies find workarounds to pay for foreign AI.

AI has not solved infra it seems. Curious to see the postmortem.

It sounds like great engineering, but everyone in the coding agent space is building the same thing. And big labs like Anthropic and OpenAI have more money to burn. Cloud agents that run in VMs are being offered from everyone from VPS and infra companies to the frontier labs.

You could definitely tighten a harness to an extent that a coding agent would only be able to read files that you directly give as context.

But the truth is, much of the utility of the model is allowing it to grep across the codebase and explore for context.

I have great interest in zero knowledge inference, but as far as we know, it is difficult to build any sort of efficient representation.

> Codex did surface permission prompts for the add, commit, and push. It didn't run them fully invisibly. So — didn't I approve this?

Operator error exists with or without AI. Installing any dev tool that could exfiltrate your information means you are responsible for securing your environment.

Use a devcontainer. You can find starting examples on the official claude code and codex github repos. Configuring a firewall script to block egress. That being said, you will likely allowlist openAI domains so I'm not certain if the sites feature will be blocked. Worth testing.

LLMs are trained on public data (as well as illegally obtained data see: Anthropic 1.5B settlement). LLMs are nothing without the huge corpus of human data that powers them. There is an argument that research of this kind should be restricted to governments and regulated universities rather than opaque public companies with competing incentives. Or research should be stewarded by genuine non-profit collectives with democratic leadership. e.g. like internet standards, telecom, etc

I do not think we can trust private companies, no matter the virtue signaling they put forth into the world, to effectively regulate themselves and inform the public and scientific communities about risks. Their ongoing conflict of interest poses serious credibility risks.