> if the temperature is fixed at a value where the top logit is always picked, then because they weights are fixed, the exact same input should produce the exact same output. Am I missing something?
Yes.
Your input is part of a batch, and you don't know where in the batch it is. By default batches are not invariant and VLLM only supports invariance at all on some Huwaei Ascend hardware.
Yes.
Your input is part of a batch, and you don't know where in the batch it is. By default batches are not invariant and VLLM only supports invariance at all on some Huwaei Ascend hardware.
See https://docs.vllm.ai/projects/ascend/en/latest/user_guide/fe...