Veedrac

This is falsely assuming that attention is the primary bottleneck. It's not. If you redirected £50B into solar, wind, and battery subsidies, you would end up with straightforwardly more green energy. Solar and wind are not resource-bottlenecked technologies. You can just buy more.

It's a large niche because it was unexploited before us, but our niche is in the resources and strategies accessible to generalized intelligence.

Humans used to have other niches, when there were resources differentially accessible to persistence hunting, but that niche basically no longer exists.

Humans of course aren't in _every_ ecological niche. I agree with you there. AI _could_ occupy only the ones we're not in, if we somehow found some stable steady-state that constrained it that way.

I don't think this invalidates the worry in the slightest.

Humans have one ecological niche. Soon we will have zero. That is worth worry.

I thirty (random long word, letter pairs) on a free model and it failed. I tested a SOTA model and it passed flawlessly. In both cases I denied tool use and spelling words out. So it seems obviously false that AI can't get better.

It's pretty wild that AI has solved a Millennium Prize problem and can accurately multiply two 40 digit numbers without tools and we still get stochastic parroting of claims like this.

Simple way to imagine this in your head.

Hold two ping pong balls at arms' length, one above and further away than the other. Right now you are the sun.

Put a red dot on top of the lower one, and then rotate it just slightly until you can't see it. This is the observer on Earth, who is after sunset.

Put a red dot directly in the middle of what you can see on the upper ball. This is the center of where the sun is striking the moon.

Keep looking at this dot on the moon. Now, keeping their relative positions fixed, bring the balls towards you and up, until you are looking from the perspective of the observer on Earth. What happens to the dot on the moon? It appears to rise up away from you.

That's the paradox.

How a good person writes a post on a topic like this:

> Some people are uncertain whether [subject] is a moral patient. Fortunately, they are not, which we know because [strong arguments about the nature of consciousness].

How an evil person writes a post on a topic like this:

> Beware that some people think that [subject] could be a moral patient. This is nonsense, because if they were a moral patient, we would have to respect their preferences. Anyone trying to convince you otherwise is trying to take your status away. You can dismiss them by pointing out that [subject] is [aspect in which subject is not identical to the speaker].

A classically pretrained LLM does not have a concept of having determined its previous tokens — it has only ever observed inputs that it had no causal influence over. This is why it's valid to say its actions are predictive and not determinative.

No, those aren't synonyms at all.

If I steer a car to avoid a predicted collision with a wall, this is not me 'predicting' the car. I am steering the car based on a prediction.

The distinction is that it's not 'predicting the next token'. Instead it's _determining_ the next token based on a prediction of its reward signal.