Comment Comment by BoorishBears Not surprising it's a hard cutoff: they almost certainly have two infrastructure configurations for the two max sequence lengthsFewer nodes dedicated to prefill per instance, and fewer nodes in total since you don't need to support a higher KV cache.Disaggregated inference also means they can tune the balance of compute dedicated to prefill seperately from decode parent BoorishBears 3 months ago
Fewer nodes dedicated to prefill per instance, and fewer nodes in total since you don't need to support a higher KV cache.
Disaggregated inference also means they can tune the balance of compute dedicated to prefill seperately from decode