ianlpaterson

I built this to answer a question for myself: which model should I actually route each type of task to? The harness runs 38 deterministic tests (CSV transforms, letter counting, modular arithmetic, regex extraction, code gen, multi-step instructions), costs $2.29 per full run across all 15 models, and all scoring is programmatic. No LLM judge for primary scores.

The surprising part was the QA process. My initial results showed Haiku beating Sonnet. That turned out to be a json_array scorer bug where max_score was set to expected_row_count instead of len(expected_rows), producing quality scores above 100%. A thin-space Unicode character (U+2009) in Gemini Flash responses broke three regex scorers silently. I ended up running 5 separate QA passes, each using a different model, and each pass found bugs the previous ones missed.

Gemini 2.5 Flash scored 97.1% at $0.003/run w/ a 1.1s median response time. Opus scored 100% at $0.69/run. GPT-oss-20b scored 98.3% for $0. The cost spread across models that all score above 95% is genuinely hard to justify for most tasks.

Scoring code and raw results are in the post. Happy to answer questions about methodology.

Concur - there are things I've been wanting to get to for (checks calendar) a decade+? that I'm now shipping. It's awesome.

There's an intermediate step, which is to use a combination of claude code sandboxing (bubblewrap), plus some pre tool hooks to look for sketchy commands, but it's still interactive and probably not the right longterm approach.

I've been running Claude Code as my primary dev environment for ~6 weeks now, and the planning/execution split is real but undersells the bigger insight: file-based memory is the operating system layer that makes it work.

Claude Code is stateless between sessions. Every /compact or context window overflow wipes the slate. The fix is a CLAUDE.md file that acts as a boot loader, plus a memory/ directory tree organized by topic. The agent reads its own instructions on startup, writes lessons when it makes mistakes, and loads topic files on demand.

The result is an agent that gets better over time despite having no persistent state. I've got ~200 condensed lessons across 10 topic files, and the error rate on repeated tasks has dropped noticeably.

The separation of planning and execution is step 1. The separation of ephemeral context and durable memory is step 2, and that's where the compounding returns live.

yes!

First impressions of Qwen3.5 35b are that it's amazing

Coming from OpenClaw, it's pretty amazing how fast pi is, particularly paired with Qwen3 that dropped today. It's a magical time.

Unclear, how much is too much... what?

Author here. I use Claude Code as my primary dev environment. Wanted autonomous agents I could message from Slack without sending everything through cloud APIs.

The 14 errors are the real content. OpenClaw's docs assume you're using hosted models. Running a 24B local model on a Mac Studio exposed silent failures at every layer: model registry, auth resolution, context management, sub-agent spawning. Error #14 required patching OpenClaw's compiled JS bundle directly.

Key optimization that made it usable: KV cache quantization (Q8_0) nearly doubled usable context and got generation speed from 12 to 49 t/s. Dual-slot GPU allocation prevents sub-agents from starving the main agent.

Happy to answer questions about the setup.

The security concerns here are real but solvable with the same discipline we apply to any privileged software.

I run OpenClaw on Apple Silicon with local models (no cloud API dependency). The hardening checklist that actually matters: run the gateway in userspace, bind to loopback not 0.0.0.0, put it behind Tailscale or equivalent - and don't put sensitive data or let it access sensitive systems!

Session bloat is the other real risk nobody talks about - vague task definitions cause infinite tool-call loops that eat your entire context window in hours, which could be expensive if you're paying per API call.

The "dangerous" framing conflates two different problems: (1) users giving agents unrestricted access without understanding the blast radius, and (2) agents being deliberately weaponized. Problem 1 is an education gap. Problem 2 exists with or without OpenClaw.