Hi, I had a quick question. Would it be correct to say the following?
1. For long inputs and short outputs, the inference can be arbitrarily number of times faster, as it avoids repeated KV computation.
2. Conversely, for short inputs and long outputs, it might be slightly slower, since loading and storing the KV cache are on the critical path of the execution.
1. For long inputs and short outputs, the inference can be arbitrarily number of times faster, as it avoids repeated KV computation.
2. Conversely, for short inputs and long outputs, it might be slightly slower, since loading and storing the KV cache are on the critical path of the execution.