Author here. Want to share the results of my experiments with local inference. I failed to get decent output from my previous setup (https://news.ycombinator.com/item?id=48587865). But this stack is performing much better.
The local stack only became reliable after fixing multiple separate failures in the stack, fixing the coding harness, improving system prompt and other tweaks in different components.
Concrete repo evidence in the post is from Clawrium(https://github.com/ric03uec/clawrium) PRs #855, #859, and #861. Will continue to iterate on this till I can offload about 30% of the project work completely to the local models.
The local stack only became reliable after fixing multiple separate failures in the stack, fixing the coding harness, improving system prompt and other tweaks in different components.
Concrete repo evidence in the post is from Clawrium(https://github.com/ric03uec/clawrium) PRs #855, #859, and #861. Will continue to iterate on this till I can offload about 30% of the project work completely to the local models.