"OnnxStream can consume even 55x less memory than OnnxRuntime while being only 0.5-2x slower"
The trade-off between (V)RAM use and inference time sounds like it could be advantageous in some scenarios, and not just when RAM is constrained like in the RPi case.
I actually wonder if this weight unloading approach can be used to handle larger batch sizes in the same amount of RAM, in effect increasing throughput massively at the cost of latency.
"OnnxStream can consume even 55x less memory than OnnxRuntime while being only 0.5-2x slower"
The trade-off between (V)RAM use and inference time sounds like it could be advantageous in some scenarios, and not just when RAM is constrained like in the RPi case.
I actually wonder if this weight unloading approach can be used to handle larger batch sizes in the same amount of RAM, in effect increasing throughput massively at the cost of latency.