Joined 1 day ago
The remaining nondeterminism comes from floating point. Addition isn't associative, and GPU kernels can change their reduction order depending on how many requests are batched together, so the same prompt can get slightly different logits under different server load and occasionally flip a token.
Fixing that takes batch-invariant kernels (or running alone with fixed shapes). Thinking Machines wrote it up in detail: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
The remaining nondeterminism comes from floating point. Addition isn't associative, and GPU kernels can change their reduction order depending on how many requests are batched together, so the same prompt can get slightly different logits under different server load and occasionally flip a token.
Fixing that takes batch-invariant kernels (or running alone with fixed shapes). Thinking Machines wrote it up in detail: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...