It is currently faster than the stock Candle / MPS shaders it replaces on MacOS/ARM64, and IIRC a bit slower than OpenVINO/CPU on my old Windows laptop, where I never got OpenVINO/GPU to compute correctly. Candle didn't have support for GPUs on MacOS/Intel, and OpenVINO ceased to be supported there.
Compared to OpenVINO (I tried ONNX runtime too, but never got it produce correct outputs with my quantized models) it is very nice to be able to build the exact kernels I need, at the quantization settings and precision that works for the models I have and with the custom operators required (speech models do a lot of non-standard stuff), run from a single set of sources, and not have to ship a hefty third-party DLL, and having to deal with their memory leaks and other stability issues.
Compared to OpenVINO (I tried ONNX runtime too, but never got it produce correct outputs with my quantized models) it is very nice to be able to build the exact kernels I need, at the quantization settings and precision that works for the models I have and with the custom operators required (speech models do a lot of non-standard stuff), run from a single set of sources, and not have to ship a hefty third-party DLL, and having to deal with their memory leaks and other stability issues.