My first version used plain `mmap`. On the 8 GB M2, a cold 3.36 MB expert took 10 ms with mmap and 2.8 ms with `pread`. The full simulation was 0.50tok/s for `mmap` vs 4 tok/s for `pread`
With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work
And common weights still use mmap for simplicity
So, I believe llama.cpp might run it under 2gb, but I assume it will be slower
With `mmap`, OS loads pages reactively as the model touches them. It doesn’t know which experts were selected or when their reads could overlap with GPU work
And common weights still use mmap for simplicity
So, I believe llama.cpp might run it under 2gb, but I assume it will be slower