Well, looking at Triton Inference Server + OpenVINO backend [1]...uff... as you said: "significant amount of development effort". Not easy to handle when you do it first time.
Is ONMX runtime + OpenVINO [2] a good idea ?
Seems easier to install and to use: Pre-built Docker image and Python package...
Not sure about performance (the hardware-related performance improvements - they are in OpenVINO anyway, right?).
Is ONMX runtime + OpenVINO [2] a good idea ? Seems easier to install and to use: Pre-built Docker image and Python package... Not sure about performance (the hardware-related performance improvements - they are in OpenVINO anyway, right?).
[1] https://github.com/triton-inference-server/openvino_backend
[2] https://onnxruntime.ai/docs/execution-providers/OpenVINO-Exe...