Just a bystander but you just quoted this:
> My whole premise is that it doesn’t have better than human intelligence. Everyone glosses over how we go from ChatGPT 4 failing its own unit tests to geometrically self-improving hyper-intelligence.
Then did exactly what the quote is talking about, assuming we achieve better than human intelligence.
> But you seem to think there's some barrier that means that even if it was a better-than-human programmer (implicitly for general-intelligence reasons), it wouldn't be able to geometrically self-improve.
The question is how exactly do we go from a below human intelligence to an above human intelligence. Without a clear path to that reality, making trade offs in order to protect against low probability worse case scenarios which require it doesn’t look like a good deal.
The reality that right now human level intelligences have a hard time even keeping these below human intelligence systems operating seems like a useful checkpoint. Maybe we can hold off on doing distasteful things like consolidating control until they can at least wipe their own bottoms as it were.
Then did exactly what the quote is talking about, assuming we achieve better than human intelligence. > But you seem to think there's some barrier that means that even if it was a better-than-human programmer (implicitly for general-intelligence reasons), it wouldn't be able to geometrically self-improve.
The question is how exactly do we go from a below human intelligence to an above human intelligence. Without a clear path to that reality, making trade offs in order to protect against low probability worse case scenarios which require it doesn’t look like a good deal.
The reality that right now human level intelligences have a hard time even keeping these below human intelligence systems operating seems like a useful checkpoint. Maybe we can hold off on doing distasteful things like consolidating control until they can at least wipe their own bottoms as it were.