Apple was running a model double the size of the available memory. Not sure if that was a sweet spot they found or if you could sacrifice response time to run even bigger models.
The paper is worth a read in full as what they are doing is pretty cool:
"Then, we introduce two complementary techniques to minimize data transfer and maximize flash memory throughput:
• Windowing: We load parameters for only the past few tokens, reusing activations from recently computed tokens. This sliding window approach reduces the number of IO requests to load weights.
• Row-column bundling: We store a concatenated row and column of the up-projection and down-projection layers to read bigger contiguous chunks from flash memory. This increases throughput by reading larger chunks."
The paper is worth a read in full as what they are doing is pretty cool:
https://arxiv.org/pdf/2312.11514
Highlight from the paper...
"Then, we introduce two complementary techniques to minimize data transfer and maximize flash memory throughput:
• Windowing: We load parameters for only the past few tokens, reusing activations from recently computed tokens. This sliding window approach reduces the number of IO requests to load weights.
• Row-column bundling: We store a concatenated row and column of the up-projection and down-projection layers to read bigger contiguous chunks from flash memory. This increases throughput by reading larger chunks."