carloslfu

what does your company do?

It wasn't either/or, the N-gram table is part of Qwen itself and stays on disk. I’ve now added its 1.5GB MTP draft head too, it gets 86% acceptance and about 1.24× faster decoding on my 48GB Mac.

thanks! in part, I was wondering how you got the code into files. I guess you copy pasted it inside a file, am I right?

This is next! in the works rn.

thanks!

> ditched oLlama"

yeah! this is interesting.

> 8.1GB per slotserve process is a lot! Is that in your control?

yes, it is hard, but I agree the smaller the better. I'll work on that

> If it's local and open-weight, this could be marketed this way I think.

I like this!

> what's the actual, real use case for slotserve?

I'm working rn on an app on top of it that closes the loop and is a fully local AI app, an experiment. I'll publish it as soon as it is usable!

> built a small html hello world served via Python

What did you use as a harness here?

great idea!! a native app would be awesome

Both projects are different in scope. Think of slotstream as optimizing for memory and for this specific model for now, my intention is not to build an inference engine the same as oMLX

Interesting! I'll check it out

true

for the record, I'm almost 35