Comment by rocauc

The model supports batch inference, so all prompts are sent to the model, and we parse the results.