Comment by dada216

I have similar hardware and requirements. Spend the time to write a test process to try multiple quantization configs/multiple inference engine and throw at it a bunch of request you can replay and evaluate the results of. We got it down to a process and everytime there's a new model/options/inference engine version we just run the internal bench/eval suite and make a choice.