qwesr123
Joined 1 year ago
Karma
328
Submissions
15
Member since
Jun 13
Recent submissions
Comment · 8 months ago
How do you plan to be harness agnostic if you design requires using the agent SDKs (Claude Agent SDK, etc.)?
Comment · 9 months ago
FYI the MarginLab Claude Code degradation tracker is showing a statistically significant ~4% drop in SWE-Bench-Pro accuracy over the past month
Comment · 9 months ago
Thanks! Daily confidence intervals are quite large and not super useful at the moment. Weekly aggregation is more sensitive. Hoping to increase sample sizes but it is quite expensive! Would be about $100-$150/day in API costs. We are using the Pro x20 subscription ($200/month).
Regarding more subtle degradation tracking, it is on the roadmap.
Comment · 10 months ago
Where are you getting SWE-Bench Verified scores for 5.2-Codex? AFAIK those have not been published.
And I don't think your Terminal-Bench 2.0 scores are accurate. Per the latest benchmarks: Opus 4.5 is at 59% GPT-5.2-Codex is at 64%
See the charts at the bottom of https://marginlab.ai/blog/swe-bench-deep-dive/ and https://marginlab.ai/blog/terminal-bench-deep-dive/
Comment · 10 months ago
Not sure why they don't compare with others, but they are actually leading on the benchmarks they published. See here (bottom) for a chart comparing to other models: https://marginlab.ai/blog/swe-bench-deep-dive/