frotaur

This was not a neuron. This was from a trained sparse auto encoder.

In a few words, they found a way to map residual stream activations (so, a vector) into a few interpretable sparse concepts. They also had the converse maps, that translate concepts into activations.

Golden Gate claude was obtained by artificially increasing the activations in the direction of the 'Golden Gate' concept. It was not neuron surgery.

Indeed, I see no difference between a chinese model hosted in Europe vs a Europe-grown model.

You could argue that potentially you can train a backdoor into a foreign-grown model (that has been demonstrated to be possible in toy examples), but I would highly doubt that this would be the case today, given the potential reputational damage if discovered.

I think you are free not to care about what they are doing. It turns out that big O notation has some useful properties, and in particular it is a useful definition as it allows you to define complexity classes which are a very useful tool for analyzing algorithms in general.

Much of our 'practical' understanding of algorithms also in part came from theoretical advances.

What makes you so certain that this 'way of developing software' cannot eventually be understood and practiced by AI?

I think precisely the premise of the blog post is that there will be a moment (sooner or later) where flexing that 'engineering' muscle to avoid it disappearing is irrelevant, in that this muscle will not be needed due to AI replacing it.

So it seems you are rejecting this premise?

I think this take is extremely cynical and made in bad faith. Lots of dangers that are due to AI acceleration are raised by a very wide mix of experts, scientists, engineers and researchers.

It is clearly incorrect to call it a conspiracy theory. You can be skeptical, as there is no clear consensus even among experts, but outright dismissal is simply an unscientific position.

The huggingface incident was reviewed by independent researchers, which explicitely declined any payment from OpenAI tonpreserve their integrity. They work for non-profits concerned with AI safety.

They claim that what happened was very much not because they were 'carefully engineered and instructed to do those things'.

Similarly, some wikis which were hijacked by agent to be used as messageboard were actually not disclosed by OpenAI (probably trying to conceal, as website showed likely activity from OpenAI researchers visiting the site after the incident) and discovered independently.

I don't know how you can claim that this was still on purpose by OpenAI as some sort of publicity stunt.

It is a fact that among experts there is no consensus on saying '(super)intelligence is broadly safe and easy to control'. There might even be a consensus forming on the opposite claim.

Regardless, why would there be no scientific consensus if the question was easy and clear cut? I think the easiest reason is that these are hard questions to answer.

that's essentially it, if the proof is incorrect it does not compile which signifies a problem in some step.

In most AI takeover scenarios, if you take as a premise that the AI has human or above-human intelligence, and that it is misaligned, it is obviously aware of the pull the plug possibility.

Therefore, as you would if you were in its position, it will plan around it. For instance, by acting perfectly aligned for 2/3 years, continuing the improvement of its capabilities while being deployed in ever more systems.

Once it's confident it can act with high probability of success, it would then turn on us. This phenomenon is called 'treacherous turns'.

Any scenario in which you assume you have ASI or AGI but also find a 2-sentence way to foil the AI's plan is inconsistent, as the AI will also have thought of this failure mode.

There is no intrinsic physical reason that the difficulty of 'defending' a system is symmetric with the difficulty of 'attacking' it.

For instance, it is not because you are able to design and release a (biological) virus that you are also able to defend from it (design a vaccine and inoculate the world's population).

If we're lucky, it might be the case for many instances of problems, but there is no a priori guarantee that this holds.

The point of lean proofs (as it stands) is simply one bit of information: that a given mathematical statement is indeed true.

It's a way to be absolutely certain (modulo bugs in the lean kernel) that a proof you came up for a statement is indeed correct. It is really not meant to be analyzed, much less now that they are fully llm written.

Not sure about the dystopia... Had a similar thought when covid was beginning 'wow pretty exciting, just like in the movies'.

Turns out actually living some terrible catastrophe is only fun in the movies.

I fail to see why this is a good argument against AI replacing humans, given that humans also produce errors.

Can you specify why we should see things differently if the behaviours the agents display are driven by parsing LLM responses and executing commands?

The energy to make the cotton bags presumably also involves lots of drilling and extracting, at least currently.