Comment by AnthonBerg

I'd give it another go!

For what it's worth, I could squeeze latency down to sub-millisecond buffers on Linux on an AMD AM4 machine. With random USB audio devices.