Comment Comment by Roxxik I have prototyped something similar with ollama some months ago.Do you mmap or issue reads on demand? Also do you use io_uring to interleave compute with io or do you spawn extra threads?I also tried predicting which experts get reused and I managed to beat a simple LRU very slightly.EDIT: That was on Kimi 2.5 but even worse quant than 4bit. IIRC it was 2.6 or so parent Roxxik 4 months ago
Do you mmap or issue reads on demand? Also do you use io_uring to interleave compute with io or do you spawn extra threads?
I also tried predicting which experts get reused and I managed to beat a simple LRU very slightly.
EDIT: That was on Kimi 2.5 but even worse quant than 4bit. IIRC it was 2.6 or so