Comment by DavidSJ

Whether you count tokens or sequences, it’s about 25x the usual batch size. My guess is it makes for a fancy benchmark but isn’t actually useful. Would be interested in being proven otherwise.