Comment by boroboro4

I think the biggest architectural change here is them doing different compute for prefill & decode, with pretty much architecture from this microsoft research work from 2024 https://arxiv.org/abs/2405.05254, very exciting stuff!