tiarafawn

Could the real reason for this be more centered around generation and control of new training data?

I suspect the same for the forced high AI usage quotas for developers at MS etc. We've had multiple generations of models trained on all of the code that's available and there are diminishing returns on how much that data can do for training now. Newly published publicly available data is also made up of a significant portion of slop.

The best way to get fresh training data from real human brains might be to have real humans use your first party tools where you control all of the telemetry.

I call this getting slopped in the face

> if AGI (artificial general intelligence) or superintelligence do in fact one day arrive, it might not seem like much of a leap at all.

That does not seem like a valid conclusion to draw from the observations in the article.

If superintelligence can be achieved, I'm pessimistic about the safe part.

- Sandboxing an intelligence greater than your own seems like an impossible task as the superintelligence could potentially come up with completely novel attack vectors the designers never thought of. Even if the SSI's only interface to the outside world is an air gapped text-based terminal in an underground bunker, it might use advanced psychological manipulation to compromise the people it is interacting with. Also the movie Transcendence comes to mind, where the superintelligence makes some new physics discoveries and ends up doing things that to us are indistinguishable from magic.

- Any kind of evolutionary component in its process of creation or operation would likely give favor to expansionary traits that can be quite dangerous to other species such as humans.

- If it somehow mimics human thought processes but at highly accelerated speeds, I'd expect dangerous ideas to surface. I cannot really imagine a 10k year simulation of humans living on planet earth that does not end in nuclear war or a similar disaster.