Comment by jboss10

I think the "architectural features" are part of the model, not the kv cache. So implementing it would be difficult and expensive.