breuleux

> I don't think there are actually that many domains where you can't simulate them with enough knowledge and compute.

The problem is that you have to simulate them accurately enough and fast enough. A perfect simulation of physics will necessarily be orders of magnitude slower than physics. To accelerate it, you need to figure out which parts can be cut or approximated without derailing the model, and just that part will require a lot of back and forth with reality.

Success isn't guaranteed either. If a physical system does whatever it does optimally enough, fast simulation is just physically impossible. Furthermore, even simple chaotic systems require unrealistic precision in the initial parameters to be useful. Natural entities are fairly chaotic and also pretty darn efficient at what they do, so it's plausible that even the best possible simulation of a middle manager is a slower bottleneck than a real middle manager.

It's also unclear how intelligence actually scales in practice. A greater intelligence would be better at researching intelligence, but also harder to improve, so why would it explode rather than go up steadily? What are the actual resource requirements of various types of intelligence like persuasion or embodied intelligence? Coding, translating, copywriting, could very well be low-hanging fruits, very easy tasks we're very bad at, which doesn't say much about the hard tasks we're good at (embodied/animal intelligence, most likely).

There is also the problem of alignment, which is verifying that version N+1 behaves in a way that version N approves of. This is ostensibly the problem that we humans have with respect to AI development -- how to make sure they align with our values -- but it's also true of AI with respect to its own successors. Improving oneself without losing the plot would necessarily be slower than throwing stuff at the wall like evolution does.

The way I see it, if you made a large simulated world where virtual beings evolved and where each generation took a millisecond from birth to death, you would see extremely rapid evolution, but you wouldn't be particularly worried about these beings breaking out of the simulation and taking over the world, because fitness in the virtual world doesn't necessarily translate into fitness in the real world. If you create stuff within a toy world, you get toys.

I think the same is true of "research" and intelligence in general. There is no such thing as a general intelligence or a general capacity to research. It's always fine-tuned to an ambient setting, including ours. Even within humanity, our intelligence is most effective in fields where rules are rigid and quick and decisive feedback is available (math, engineering, computing), but it is much less effective in soft sciences, and disappointingly useless in politics. In other words, the dynamics of innovation and improvement are not purely mental, they are very much tied to external factors that are not always easily controlled.

> it'd also require "physical intuition" to be a harder to generalize skill than research, which also seems like it can't possibly be true - one of these is just physics

It's not "just physics", it's a honed system for surviving a realistic range of physical conditions, assess material properties on touch, adapt manipulation to arbitrary shapes, predict trajectories, take into account variations in ground, smell, wind, and so on; not to mention the machinery of the body itself, which self-repairs and protects us against a range of complex threats.

The point isn't that machines couldn't do all this. The point is rather that you can't quickly iterate and adapt designs to do it from an ivory tower. Pure "research" simply won't get you there, you also need to experiment, and insofar that physics limit the throughput of these experiments, the speed of improvement will be throttled severely. It is a mistake to think that "RSI" could break this, because it is categorically the wrong lever to pull.

Extrapolating on the evidence that AI appears to have jagged intelligence, it is entirely possible (probable IMO) that ASI will plateau far above human capability in digital and symbolic intelligence, while staying far below in the ability to operate in the physical world.

As impressive as AI might be at virtual tasks like coding, virtual reality is remarkably different from physical reality in ways that artificially inflates AI results. One, it is far simpler. Two, the feedback loop is far, far quicker. It's possible AI could get the animal intelligence and animal body that lets us humans actually apply our intelligence in reality, but nothing in current technologies really indicates that this is a given.

AI doesn't need to be smart to seize our resources. They need brute strength, a proper physical intuition, and "hands". That may very well be several orders of magnitude harder than anything they're currently doing, we're just lucky it was in our starting build.

The best problems to work on are not necessarily the hardest ones, nor the ones that need the most intelligence. They're the problems you, or other people, actually have. Are you going to give up on painting your deck because it's too easy and you don't need a 500x genius to do it?

Yeah, it's a bit absurd on its face to speak of equilibrium in the context of evolution, which is basically a litany of innovations and equilibrium breaks. Nature regularly throws itself out of whack, like when cyanobacteria appeared I recall, and then when nature made us, obviously. Whatever equilibriums existed through time are rarely returned to once broken.

> if we disrupt the system, things will eventually go back to that equilibrium. And if we don't see it, we just needed to wait longer.

Even if that's true, quite a bit rides on how long "longer" is. If "longer" is a million years, that's not exactly good news.

My criteria for originality may be more lenient than yours. I think it would be difficult to find sci-fi that isn't derivative of older ideas to an extent, and it's harder for more recent work to stand out, because basically everything's already been done. Managing to combine a dozen ideas from previous works into a coherent whole is far from uncreative. So I feel the author was original enough in his approach, even though I thought it was scientifically inept.

I increasingly feel like "consciousness" is little more than "that thing we humans have... you know what I mean... you have it too!" and that any attempt to elevate it to a more general principle is doomed to failure. Sure, we can easily extend it to other living beings because we share a rich evolutionary history, but beyond that...

As far as AI goes, I think that whatever it is that they have is analogous in significant ways, but also irreconcilably different in significant ways. They aren't proceeding in a straight line towards consciousness either, because why would they? They're doing their own thing which happens to produce similar outputs. It's a problem in terms of value: we value consciousness, because we value ourselves, because we value our output. If it turns out consciousness as we know it is just a minor implementation detail with no cosmic or long term significance, then irrespective of whether AI is conscious or not, we still have to grasp with the challenge of explaining why consciousness is valuable in the first place if not for its outputs.

I'd argue that the reason why we value life, consciousness, the ability to suffer, was always arbitrary, and it relied a lot on the unicity of human experience. It was never robust to the breaking down of human capabilities into inspectable, reusable, upgradable and resettable modules. Or the radical decoupling of capabilities we hitherto considered correlated, like writing about love and feeling love, and many other things.

I only read the first book of the trilogy and the "science" in it was so profoundly absurd that I couldn't motivate myself to read any further. I think part of the issue is that it had been sold to me under the guise that it was hard sci-fi, and it was anything but.

I disagree it's unoriginal, though. It's pretty original, you just have to be prepared to suspend your disbelief (it's kind of on me that I wasn't when I read it).

I think any alternative system has to figure out how to attract and reward the kind of people who are intrinsically motivated by truth, discovery and knowledge, and repel the kind of people who aren't.

The chief issue with external incentives like "fame" or "money" is that they will attract a lot of people who care about the reward, but not the work. If they can find a way to cut corners, they will. So it works fine if you have a very tight, objective and reproducible evaluation criterion such that you can't get rewarded for poor or sloppy work, but even in science this is difficult to establish, especially in a crowded field where most advancements will be small and incremental. Science works a lot on trust, if only trust that the described experiments were indeed made and the results were indeed as reported.

That's why I feel we get the best science when it's driven by people who just intrinsically want the best science, because getting it right is a reward in and of itself. It'd be neat if it was possible to identify them and just let them cook, regardless of whether results are produced or not. Paradoxically, I think this may be easiest in a setting where academia is undervalued -- the knowledge-seekers will always self-select into research (provided the conditions are good enough and they are not too undervalued), but the fame-chasers will have to find their validation elsewhere.

Sure, but is the limiting factor intelligence, here? Give human intelligence to dolphins and they're still not going to be able to pave the world. In order to remake the physical world, we needed certain physical abilities -- we needed the animal mind and body that took billions of years to evolve and embeds advanced priors about the world, and we needed it packaged properly. Irrespective of our intelligence, our body is a general world navigation device adapted to grab and move stuff. We use this innate tool to remake the physical world, but we didn't make it, and we'd be impotent without it.

With AI it's a bit of a chicken or the egg situation. Before it can ablate the complexity of the world, it needs to be able to deal with it, including adversaries such as ourselves. For this it needs a tool as sophisticated as our bodies, but could AI develop such a tool for itself without the help of this tool? I'm not convinced.

> Already businesses that have more compute and access to data seem to eat the world around them.

Do they? Business valuations are more a reflection of what people believe will yield returns than a reflection of reality. A lot of big tech behemoths could disappear from the face of the Earth overnight and it would make little difference to our lives.

Natural resources and energy are the actual backbone and it is a true catastrophe when these are disrupted. In comparison, disruption to compute or data is just an inconvenience.