languid-photic
Joined 1 year ago
Karma
129
Submissions
45
Member since
Apr 24
working on: https://github.com/voratiq/voratiq
get in touch: [email protected]
View on Hacker NewsRecent submissions
Comment · 6 months ago
also feels like a good posttraining task
Comment · 6 months ago
would be fun to do zig -> rust -> zig and to measure the delta
(in a VAE-ish way, kl div on the embeddings?)
Comment · 6 months ago
It’s mostly a bandwidth thing. We’ve seen the pattern consistently, but haven’t had time yet to write up the analysis carefully.
We are not the only ones to see the reasoning inversion.: https://arxiv.org/abs/2510.11977, https://arxiv.org/abs/2502.08235, https://arxiv.org/abs/2507.14417
Comment · 6 months ago
My point is more reasoning often leads to worse "scope creep/churn, codebase fit, maintainability".
Comment · 6 months ago
We also use a secondary signal from blinded multi-verifier reviews. Each verifier ranks the candidates, and those verification outcomes serves as an additional quality signal. It's somewhat similar to consensus labeling.
Btw, this also helps manage scale. Eg you have 15 diffs to review. Run a few verifiers to get a short list, then review directly and apply the best.
Comment · 6 months ago
Agreed. Harness is really important. Especially since many labs are now post-training agents directly in their native harness.
(Which is why my prior is that third party harnesses would not perform as well. But I haven't actually measured this.)
Comment · 6 months ago
Yes, the signal we are measuring is quite different from most evals.
We are measuring sth much closer to: when multiple agents compete on the same spec, which one produces the patch that holds up best in code review?
Most evals are static / synthetic, and for code, generally stop at tests. Test evals are weak proxies for quality since it's difficult to encode qualities like scope creep/churn, codebase fit, maintainability etc in tests. [1]
Almost every agent in a given run can pass tests at this point, but there is large separation during review.
Comment · 6 months ago
I agree!
So far we have been native harnessmaxxing, which simplifies things a lot.
The configuration space around open models is much larger. Eg which models, capability heterogeneity, which harness, networking, data egress / privacy, etc.
If anyone is getting very good production code out of open models, I'd love to do a user interview to better understand your setup. Email is in my bio.
Comment · 6 months ago
Yes! It depends on the extent of changes needed.
If the changes needed are small, I'll apply the best implementation as a foundation and then just iterate directly.
If the changes needed are drastic, it usually signals that there was sth wrong/ambiguous/etc in the spec (or the ensemble was too weak, which is rarely the case). In cases like this, I improve the spec and then rerun.
If it's in the middle, I'll usually apply the best and write a follow on spec.
Comment · 6 months ago
We track performance vs. the all-in cost of completing real engineering tasks, rather than cost per token. [1]
Cost per token is a bit misleading because, as others have noted, different models use tokens in different ways. (Aside - This is also why TPS isn't a great metric).
We found that 5.5 is about 1.5-2x more expensive overall. On a "Pareto" basis, we only find 5.5 xhigh worth it. At the lower reasoning levels, 5.4 still edges it out on cost/perf.
We take a spec-driven approach and mostly work in TS (on product development), so if you use a more steer-y approach, or work in a different domain, YMMV.
Comment · 6 months ago
Agreed. As alignment improves, I'm becoming increasingly bearish on sandboxing.
Version control and isolation will probably stay useful, though, more for distributed development and workflow reasons than for safety.
Comment · 6 months ago
It’s very hard to encode the properties that matter most in code in tests. [1]
Comment · 6 months ago
I think that's Pro. Regular 5.5 is 2x regular 5.4.
Comment · 6 months ago
It's 2x/token, but for default reasoning we've found GPT-5.5 uses fewer tokens overall, so net cheaper on median. [1]
(Note, that stops being true at higher reasoning levels, where our observed total cost goes up ~2-3x.)
Comment · 7 months ago
Appreciate the clarification. But, it's still not great.
To the PM behind this - developers are sensitive to this kind of thing. Just make it opt-in instead?