Because streaming only removes the decoder stack. The embeddings and lm_head stay
resident, that is 2.10 GB of the 3.32 GB peak on 8B. And the logits tensor scales
with batch x seq x vocab, not with depth.
So it goes from "whole model must fit" to "embeddings plus one layer plus logits
must fit". That is why 8B works and why I did not try 14B.
The table on the site is the normal resident path, streaming is opt-in and BETA.
Should be clearer, my fault.
So it goes from "whole model must fit" to "embeddings plus one layer plus logits must fit". That is why 8B works and why I did not try 14B.
The table on the site is the normal resident path, streaming is opt-in and BETA. Should be clearer, my fault.