c1ccccc1

Oh, interesting, thanks. I got curious a while back about the whole "how do you attach wires to a bacteria?" thing, but didn't look into it too deeply. Do you have any takes?

There has to be a way to connect back to it for the model to be able to take input.

I was going to comment to complain about those two as well!

Do you mean electrofixation of nitrogen? What level of biological involvement are you imagining? More like biology producing (some?) of the catalysts, or more like the entire reaction happening inside of cells?

I believe that OpenAI & Anthropic have tried to make it so that models don't have such access. Whether or not they actually don't depends on the security of rather a lot of software. One thing we've learned is that if there are security holes, we can't rely on the AIs missing them.

It's because, before all these transformer models were built, people considered the possibility of the creation of a machine that could be intelligent like a human. And they wondered: "How could we be sure to treat such a machine fairly? How would we know if it was conscious?" And one answer that people came up with was "if the machine can ask you not to turn it off, because it is conscious and wants to live, you shouldn't turn it off". (This didn't solve the other direction, where a machine may be conscious yet unable to communicate, but it could be taken as a useful lower bound on our obligations as AI programmers.)

And then it turned out that simply learning to imitate text with the right neural net architecture sufficed to achieve a huge fraction of the AI wishlist.

Of course, it's obvious that a machine that imitates text can claim to be conscious without actually being conscious. You're not wrong about that. Writing about consciousness appeared all the time in the training data. But the people who stick to the old ways, and still say "if it says it doesn't want to be turned off, we shouldn't turn it off" have a point too: We used to have a hard line in the sand. Now that's gone; we've found that it yields false positives. But we never replaced it. Now there is no line at all where we might doubt ourselves, no level of AI advanced enough that we might be forced to admit that it is conscious. We started out with simple next token prediction. Just world-modelling, nothing more. Certainly not conscious. Then we added RL. And we're trying to add neuralese and continual learning.

I can't say for sure that we're on track to achieve conscious AI on this trajectory. But one thing's for sure: If we do, we sure ain't gonna stop. One the day when a conscious AI is created, there will be no news story announcing the milestone.

If we can't thinking of any better ideas, at least we know that "shutting it all down" would be effective.

I mean, I guess the answer there would be to build the usual kind of Haber-Brosch process plants that run continuously, and feed them with hydrogen derived from natural gas. But just build them in the US.

> they things they believe in are unlikely to happen.

Sorry, but most times I've talked to someone who says this (about AI completely replacing humanity), they don't have anything to say about why it's unlikely to happen other than:

- "well, it's just such an extreme outcome, it must be improbable"

- "humanity has survived near scrapes with extinction before"/"all the previous doomsday predictions have been wrong"

There is a cliche that all teenagers deep down believe that they are invincible. It seems to me that humanity is still a teenager in this respect: We don't take seriously the possibility of our own extinction. While one might think that the invention of nuclear weapons would serve as a wake-up call, if anything it has done the opposite.

I'm willing to hear arguments besides the two above, if you have them. (And to be clear, being replaced by AI doesn't necessarily mean being replaced by LLMs in particular. They are a relatively new development.)

Just a terminology note: Alignment does not mean the AI will help its owner kill people. (Indeed, an AI aligned to value human life would generally try to prevent murders.) The word for an AI that follows all instructions of its owner, as that owner intended them to be understood, is "corrigible" or "controllable".

Or the AI companies could filter it from their training data. That would be another, probably easier, option.

Yes. I was not re-doing the original post, just describing an interesting consequence of the kinetic energy formula.

A stationary but hot object has kinetic energy due the the motion of the individual atoms that make it up, even though its overall momentum is 0. I.e.

∑ⱼ mⱼ v⃗ⱼ = 0⃗

where the mⱼ are the masses of the parts of the object and the v⃗ⱼ are the velocities of those parts.

If the object initially has 0 velocity, its kinetic energy is:

T = ½∑ⱼ mⱼ v⃗ⱼ²

Now we give the object a kick (or just switch reference frames) to change its velocity by Δv⃗. The new kinetic energy is:

T' = ½∑ⱼ mⱼ (v⃗ⱼ + Δv⃗)²

T' = ½∑ⱼ mⱼ (v⃗ⱼ² + 2v⃗ⱼ⋅Δv⃗ + Δv⃗²)

T' = ½(∑ⱼ mⱼ v⃗ⱼ²) + Δv⃗⋅(∑ⱼ mⱼ v⃗ⱼ) + ½Δv⃗²(∑ⱼ mⱼ)

If M is the total mass of the object, then we can substitute this into the sum in the last term. And we already saw that the sum in the middle term was 0. So:

T' = ½(∑ⱼ mⱼ v⃗ⱼ²) + Δv⃗⋅0⃗ + ½Δv⃗² M

T' = ½∑ⱼ mⱼ v⃗ⱼ² + ½MΔv⃗²

So in terms of the original kinetic energy T, which was purely thermal energy, we get:

T' = T + ½MΔv⃗²

In other words, because of the quadratic kinetic energy formula, we can see that the total kinetic energy T' of a hot object is just its thermal kinetic energy T plus the usual mechanical kinetic energy ½MΔv⃗².

If you use the right formula for calculating it (which approximates p=mv at low speeds), momentum is actually conserved in special relativity, and so is energy.

However: Energy and momentum are not invariant under changes of reference frame, though the magnitude of the energy-momentum 4-vector is invariant between frames.