Comment by 7e

The parallelism is nice, but it only helps cloud inferencers if there is an efficiency improvement. Autoregressive inference isn't parallel, but parallelism can be recovered with sufficient batching.