Comment by MrBuddyCasino

AI infra buildup is so massive that the frontier labs should be able to offer more than one level of context length to incentivize token thriftiness.

One would think compute-constrained actors like Anthropic would have done so, unless prefill isn’t really a bottleneck compared to decode?