Comment by gb2d_hn

This just can't be sustainable. Ever increasing agent activity can only move to the point where it's literally impossible for humans to have a collective awareness of how the systems that support our lives actually work. How does anyone think this is going to end well? If as humans we loose control of vital systems, when they stop working one day (and they so easily could unless the agents are perfect and 100pct aligned with the general good of humanity (how do we even define that)) we could find ourselves without the ability to run healthcare systems and power grids, that have evolved so much such complexity that no human can intervene to fix them. I do believe that you can create feed back loops where probabilistic agents can be used to create deterministic checks and balances, and solve real problems for good. But it's a wild west at the moment and that is not what's happening in so many companies, and the solutions are not yet on the horizon. Even if the 'agents can handle it' ... where does that leave us? Unable to run our societies without puting everything through frontier LLMs? I just finished a very short story by Ted Chiang - 'The evolution of Human science' it starts at the point where it's 25 years since humans contributed a scientific paper because 'meta humans' now handle all research, and communicate digitally such that humans can only get a low fidelity report of what the agents actually know, and the endeavour to understand has become like a best effort archeological like hobby for interested humans. The story assumes that the 'meta humans' will always act in humans best interests. I believe it's from 2002, it's incredibly precient (The stories of your life and others is the collection the story is in. Many of them are very relevant to modern AI actually).