I agree, but I'm less optimistic about how far from that scenario we really are. I wonder what we'll look back on as the more damaging effect of the LLM hype: the economic bubble, or the carelessness it caused in people using the systems.
The knowledge about the nature of LLMs, their strengths and weaknesses, what they can appropriately be used for and what they should definitely not be integrated into, seems woefully lacking in many places. Just think back to the recent, disconcerting stories from the military world. We started with lawyers not understanding that an LLM can and will make up fake case law. Now we're at high-ranking military personnel not understanding that LLMs can and will make up fake target identifications.
Maybe the best hope for awareness about this spreading is a gradual slope of worse and worse, but not-quite-catastrophic-yet, highly publicised incidents.
The knowledge about the nature of LLMs, their strengths and weaknesses, what they can appropriately be used for and what they should definitely not be integrated into, seems woefully lacking in many places. Just think back to the recent, disconcerting stories from the military world. We started with lawyers not understanding that an LLM can and will make up fake case law. Now we're at high-ranking military personnel not understanding that LLMs can and will make up fake target identifications.
Maybe the best hope for awareness about this spreading is a gradual slope of worse and worse, but not-quite-catastrophic-yet, highly publicised incidents.