Comment by wongarsu

A lot of research about avoiding extinction by AI is about alignment. LLMs are pretty harmless in that they (currently) don't have any goals, they just produce text. But at some point we will succeed in turning them into "thinking" agents that try to achieve a goal. Similar to a chess AI, but interacting with the real world instead. One of the big problems with that is that we don't have a good way to make sure the goals of the AI match what we want it to do. Even if the whole "human governance" political problem were solved, we still couldn't reliably control any AI. Solving that is a whole research field. Building better ways to understand the inner workings of neural networks is definitely one avenue