nasutton12

pi is a fantastic harness! they are a good default in the same way llama.cpp is. it works with everything and that is the point.

i was steering chad in the opposite direction. one model & one set of silicon -> taken to the max. swap out your CHAD_MODEL and it still runs, you just leave the drafter and the kernels behind.

fair longer is always helpful. on a 24gb mac i'm extremely constrained, but all these exercism tasks were plausible to pass at this context length.

this is in the repo in benchmarks/matrix for most of these metrics.

uv run python benchmarks/matrix/run.py setup. i'd happily take your hardware's numbers!

sounds fun, i'll try and add it.

There are a comical amount of harness design features in the early 1.x release. A full LSP integration, find symbols, batch edits, etc. Terminal bench isn't everything but I spent a couple of weeks testing different combinations to no statistical effect greater than a bare loop. It was like running uphill against what the underlying LLM wanted to do.

sorry its a notion.

sorry, its a notion not a real site.

The annoyance is real. Let's do it.

I'll try it.