ajspig1

How do you handle provider variance on OpenRouter for the opensource models? Or do you use your own hosted version to mitigate this?

And for both opensource and closed source, does the router account for provider quality, or catch it when a provider degrades?

& the nice thing about Hermes (since its open source) is you can be reasonably sure that behavior change is coming from the model and not the harness. (probably)

agreed. I'd also love to see more comparisons like this.

I do think that a well speced out prompt can be finished faster by an agent then if prompted lazily. I've also found measuring this to be challenging, since issues that come from lazy prompting can hide until its too late to pinpoint exactly what prompt introduced them. But maybe this should be the new "git blame"... gears are already turning for what evals could look like for this.

I feel validated.

also, this assumes humans are still the primary CLI consumers. With agents increasingly being the first-class users of command-line tools, building visual design tooling for terminal UIs feels like optimizing for a shrinking audience.

  I get that "communicate your thought process" or "play along with the exercise" gets offered as the fix here. But that framing bothers me too. Why should simplicity require more justification than complexity? Google Sheets is the design. The fact that it doesn't sound like engineering is the whole point.

I'm just not convinced the solution is learning to package simplicity in a more impressive wrapper.

+ Also the fact that the Memory.md file was a hindrance to the quality of output

There are so many SOTA memory claims out there. There are also very few memory benchmarks (if any) that measure "memory" realistically. Not to mention the industry can't seem to decide on what "memory" even means. We need more general-purpose long-term memory systems. We need ways to measure those. Verify they fulfill the need and are actually SOTA.