jedharris

Never mind debates about intelligence. How many actions can a system take at 99% reliability before a human must intervene? This metric is doubling every 7 months.

"Our results show that simple algorithmic improvements can enable significantly more data-efficient pre-training in a compute-rich future."

Certainly one of my criteria for a utopia would be that some of the beings in it are engaged in intriguing or exciting activities with meaningful consequences. I'd require more than Special Circumstances to meet this criterion -- they are too small a piece of the total system.

In Player of Games we see a corner of a gaming culture which partly meets this criterion but it does not have meaningful consequences outside the gaming participants (unless you count the ways the Minds use it to manipulate Gurgeh).

Maybe by this criterion utopias are impossible, since the disruption caused by exciting activities with consequences conflict too much with the optimality of the society. But I don't think anyone now can prove this would be the case.

In the Culture Series history hasn't ended, and the members of the Culture can and do leave and live in ways the Culture would find repugnant. Also there are marginal participants (such as the Mind known as Meat Fucker) who are more or less in the Culture but act in ways the Culture finds repugnant, but are not punished or controlled.

So according to your definition the Culture is not a utopia.

This is an example that supports Scott's point that people don't have world models. The people who "believe" this don't wonder how stock market continues to operate now that NYC is a wreck. Etc.

I wonder in what sense they really do "believe". If they had a strong practical reason to go to a big city, what would they do?

> it's possible for simple normal trains of thought to latch your brain into a very undesirable state.

This seems very incorrect, or at least drastically underspecified. These trains of thought are "normal" (i.e. common and unremarkable) so why don't they "latch your brain into a very undesirable state" lots of the time?

I don't think Scott or anyone up to speed on modern neuroscience would deny the coupling of mental state and brain chemistry--in fact I think it would be more accurate to say both of them are aspects of the dynamics of the brain.

But this doesn't imply that "simple normal trains of thought" can latch our brain dynamics into bad states -- i.e. in dynamics language move us into a undesirable attractor. That would require a very problematic fragility in our normal self-regulation of brain dynamics.

some pointers to the research program please?

We often weaken ourselves by thinking we have a moral duty to be tolerant, even in the face of intolerance.

What the journey from AI assistant to full-virtuality can teach us about the nature of love.