> The total number of concurrent requests represents the number of simultaneous requests moving through the pipeline at any point in time. Unlike a GPU batch, those requests don’t wait for one another to finish, they stream through the pipeline continuously.
What do they mean by “unlike a GPU batch”? It’s not a hardware limitation of GPUs right? It’s just the scaffolding to do this in software isn’t available right
What do they mean by “unlike a GPU batch”? It’s not a hardware limitation of GPUs right? It’s just the scaffolding to do this in software isn’t available right