tarxzvf

Models hallucinate plausible answers to why they did things. It might be true and it might be complete fiction.

Thousands of words and piles of equations and not a single mistake pointed out? I have zero domain knowledge here, but enough interactions with AI agents to think this is wildly crazy.

This is impressive. Yet, before you go the AGI is nigh, ask yourself a simple question: will this spiral in or spiral out? If we feed everything the model comes up with back as training data, will we get Endless Forms Most Beautiful or will we get an equilibrium?

Yes. The only force we know that achieves this is undirected, and no single part of it stays at the top for long. Contrast with superintelligence, a single entity which does not evolve but optimizes in a directed way.

Honestly astonished that superintelligence is a mainstream idea. The story it tells makes sense only if you never bothered to dig further than its surface.

- Replace 'AI' with 'God', does it still make sense?

- Exponents still take time. 2^33 to get to current world population with no hitches.

- Solomonoff / Bekenstein / Gödel - name your favorite limiting theorem.

- For any optimization method we can literally construct a learning problem that it can never successfully learn. Take it a step further and you have a communication channel where the AI listens to everything and understands nothing.

- Was any force ever able to get close to world domination? At one point in history the US had nuclear power and no one else had it. Was that edge enough?

When we get closer to manufacturing universal intelligence its more impressive incarnations will look more like countries and corporations than omnipotent deities. The problems we’ll have to face will have more to do with consciousness and human rights than with alignment. Alignment is really more about automation at the incomprehensible scale, where the clash between dimensionality reduction and Goodhart’s Law becomes absurd.

Just replace K and JavaScript with German and English to see the vacuity of this argument. Either of several possible representations can become native to one’s thinking. The question is which is a better aid in reaching some non-arbitrary goal. The only merit of K presented and emphasized here was the supposed brevity of its programs. Personally I’ve found the habitable zone somewhere that allows for more air between ideas.