Most experts don't think this is likely, just that the potential consequences are bad enough that it's worth making it less likely. It's just like how most people don't think global nuclear war is likely, but it's still worth reducing the likelihood of.
Also, the people who worry about this aren't concerned about current ML stuff going haywire. They're worried that we're one or two algorithmic breakthroughs from something that can improve itself. If the upper bound for what sort of intelligence is possible is much higher than us, we could quickly be outclassed. As Nick Bostrom says:
> Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.
If you want to delve into the best version of the AI risk argument, I recommend Superintelligence by Nick Bostrom.
Also, the people who worry about this aren't concerned about current ML stuff going haywire. They're worried that we're one or two algorithmic breakthroughs from something that can improve itself. If the upper bound for what sort of intelligence is possible is much higher than us, we could quickly be outclassed. As Nick Bostrom says:
> Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.
If you want to delve into the best version of the AI risk argument, I recommend Superintelligence by Nick Bostrom.