It's a reasonable summary, although I feel there's some fundamentals missing. I wonder how much it will make sense to a platform engineer who doesn't already have a very solid understanding of transformer architecture internals such as the Attention mechanism and its more recent variants. I don't know if you can really get this stuff without having some underlying model internals picture in your head of what the KV cache actually is and how it falls out of repeated attention computations on a continuously growing prefix, to say nothing of understanding KV cache optimizations like PagedAttention which depend very much on the details of how you compute the attention weights with matrix multiplications, softmax, etc. I certainly couldn't make sense of something like "Rather than recompute that history, the engine keeps the attention keys and values for every token in flight" without a pretty solid understanding of how Attention works, which I had to gain from other sources.
This is also a shortcoming of Philip Kiely’s otherwise fantastic book Inference Engineering; it kind of glosses over the transformer architecture details, which you need to get elsewhere. Sebastian Raschka's books are very good for that but you have to then link it all back to how to do efficient inference, which he doesn't talk about so much.
This is also a shortcoming of Philip Kiely’s otherwise fantastic book Inference Engineering; it kind of glosses over the transformer architecture details, which you need to get elsewhere. Sebastian Raschka's books are very good for that but you have to then link it all back to how to do efficient inference, which he doesn't talk about so much.