gitpusher42

The client needs to keep a messages array, add each user message, send the full array with `stream:true`, print SSE chunks until `[DONE]`, then add the response back to the array. `/reset` can clear it

There is a python example in the server docs. (https://github.com/drumih/turbo-fieldfare/blob/main/docs/OPE...)

It is non-streaming, but can be used as a starting point.

The server is still experimental and I am fixing some problems currently. But you can try to vibecode a simple terminal client around it.

If not, create an issue on Github and describe desired behaviour

Comment · 3 months ago

I think there is a limit based on MoE number of active parameters and quantisation, bytes count for active experts. But I believe we will see more project like this for different models.

Comment · 3 months ago

Uh, maybe, who knows. I am pretty bad solving leetcode, btw

Comment · 3 months ago

Yeah, gpt-oss-120b is also MoE, so the same ssd-streaming and caching ideas should work. Feel free to fork and try implementing it!

Comment · 3 months ago

Yeah, it must be exactly the same. The same weights are used, nothing skipped or pruned. But it might have differ to MLX for greedy decode because of small floating-point nums difference

Comment · 3 months ago

yeah, looks like a page cache matters a lot I tested on mine m5 pro with 8gb memory pressure, got 27t/s instead of 35t/s Someone tested on m4 max. In regular state it was 48tok/s, but 32-42 with memory pressure

Comment · 3 months ago

Apple does something similar with their latest foundation model. They process input prompt and based on results they preload required experts. Quite neat solution for the edge devices

Comment · 3 months ago

Yeah, sure! You can select different options in the app settings at the right panel, it shows how much memory it will use

For CLI and Server, use --max-context

Comment · 3 months ago

uh, tried most of this

mmap benchmark did basically page touch experiment and cold reads were much slower, unfortunately (10ms vs 3ms)

I tried MADV_WILLNEED, F_RDADVISE and preadv. preadv reduced parallelism because requested experts are rarely adjacent in the file.

pread is still the fastest. And I think Flash-Moe got the same result too

Comment · 3 months ago

afaik there is some research at this area. Also the new apple foundation model uses related idea. they process the whole prompt and based on prompt load required experts and use only these experts for generation. It doesn't require fitting full model into memory or per token ssd streaming