anuarsh

Good question, need to research this one

I haven't tested on Apple machines yet, but gpt-oss and qwen3-next should work I assume. Llama3 versions use cuda specific loading logic for speed boost, so it won't work for sure

Thanks! I don't have much experience with diffusion models, but technically any multi-layer model could benefit from loading weights one by one

1tok/2s is the best I got on my PC, thanks to MoE architecture of qwen3-next-80B. gpt-oss-20B is slower because I load all single layer experts to GPU and unpack weights (4bit -> bf16) each time. While with qwen3-next I load only active experts (normally 150 out of 512). Probably I could do the same with gpt-oss.

CPU is much slower than GPU. You can actually use both by offloading some layers to CPU as o.offload_layers_to_cpu(layers_num=12). It is faster to load from RAM than from SSD.

We are talking about 100k context here. 20k would be much faster, but you won't need KVCache offloading for it

Absolutely, there are tons of cases where interactive experience is not required, but ability to process large context to get insights.

Hi everyone, any comments or questions are appreciated