Isn’t input so much cheaper because it’s just less work? Prefill is usually 10-20x as fast as generation.
If you have 200k context, but only 200t/s prefill, then you’ll wait multiple minutes when continuing an average session from yesterday. Or compaction - might take 15 minutes.
If you have 200k context, but only 200t/s prefill, then you’ll wait multiple minutes when continuing an average session from yesterday. Or compaction - might take 15 minutes.