It's because speech to text, text to speech, and LLMs all have latency. They are working on it. There were rumors that Apple is spending a million dollars a day on its secret AI. Probably some form of it will go into Siri.
Edit: you can try the gpt4-enhanced Bing, it works pretty well with voice
Edit: you can try the gpt4-enhanced Bing, it works pretty well with voice