The fact that batching is faster does not inherently imply some sort of inefficiency, but rather is indicative of the fact that sequential memory access is faster than random.
I am curious what the basis behind the idea that Python is the performance bottleneck for inference is.
I am curious what the basis behind the idea that Python is the performance bottleneck for inference is.