> Aggregate token throughput is about 30% lower (122 vs 170 tok/s at 16 users)
Is that actually the aggregate throughput? 8xB300 (with 4TB/s/GPU bandwidth) is only pushing 8 tg/s/session? That seems… incredibly low, even for an A100B model.
Is it actually 122 tg/s per session? (1952 tg/s aggregate throughout)?
Is that actually the aggregate throughput? 8xB300 (with 4TB/s/GPU bandwidth) is only pushing 8 tg/s/session? That seems… incredibly low, even for an A100B model.
Is it actually 122 tg/s per session? (1952 tg/s aggregate throughout)?