yvbbrjdr

We actually profiled one of the models, and saw that the last GeMM, which is completely memory bound, is taking a lot of time, which reduces the token speed by a lot.

I see! Do you know what's causing the slowdown for ollama? They should be using the same backend..

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that none of my friend has this device.

Hi, author here. I crowd-sourced the devices for benchmarking from my friends. It just happened that one of my friend has this device.

We use DeepSeek v3 in prod. Works even better than GPT-4o.