Do note that a custom host runtime is still responsible for loading firmware, booting the CPU to the starting point, and compiling/loading the program to run. While the CPU is running, the host runtime makes sure that only the CPU layers run by manually skipping Qwen’s own layers.
I wish the author would be more forthcoming about these limitations. The central claim about “no external runtime” or “not a plugin” feels a little dishonest to me since this doesn’t work without some pretty invasive changes to the inference driver.
Do note that a custom host runtime is still responsible for loading firmware, booting the CPU to the starting point, and compiling/loading the program to run. While the CPU is running, the host runtime makes sure that only the CPU layers run by manually skipping Qwen’s own layers.
I wish the author would be more forthcoming about these limitations. The central claim about “no external runtime” or “not a plugin” feels a little dishonest to me since this doesn’t work without some pretty invasive changes to the inference driver.