Comment by nullc

Does this have intelligent expert handling for high parallelism MOE? You can get very high throughput for highly parallel MOE if you can mix different queries at each expert stage, but if the batch has to run together for the whole pipeline you get a parallelism loss instead of gain.