gcr
Joined 17 years ago
Joined 17 years ago
I think you got the conversation into a state where any answer would be equally likely to feel profound to you.
Do note that a custom host runtime is still responsible for loading firmware, booting the CPU to the starting point, and compiling/loading the program to run. While the CPU is running, the host runtime makes sure that only the CPU layers run by manually skipping Qwen’s own layers.
I wish the author would be more forthcoming about these limitations. The central claim about “no external runtime” or “not a plugin” feels a little dishonest to me since this doesn’t work without some pretty invasive changes to the inference driver.
It's thought to work "better" but score calibration is still a problem.
EDIT: If you scroll down to the commit graph and have your mouse inside the big commit graph rectangle, you can't scroll up anymore!