Comment by chillee

For latency-bound inference (i.e. one request) you don't need tensor-cores since all your operations are just matrix vector multiplications.