gronky_
Joined 3 years ago
Karma
1,011
Submissions
83
Member since
Dec 6
Recent submissions
Comment · 6 months ago
Mobile rendering of the post has some issues. Tables are overflowing and not responsive for example
Comment · 8 months ago
Don’t know about other claws, with NanoClaw the agent can only rewrite code that runs inside the container.
You can see here that it’s only given write access to specific directories: https://github.com/qwibitai/nanoclaw/blob/8f91d3be576b830081...
Comment · 9 months ago
True. There’s a setting for Claude code though where you can add apiKeyHelper which is a script you add that gets the token for Claude Code. I imagine you can use that but haven’t quite figured out how to wire it up
Comment · 1 year ago
Keep in mind that this isn’t about users - the top agents on the leaderboard aren’t running an actual product on the benchmark.
If they are running their production product as is, then of course whatever is built into the product is fine.
Comment · 1 year ago
This ok from your perspective then?
def make_pass@1_agent(agent, n):
def retry_agent(problem):
for attempt in range(n):
result = agent(problem)
if result.success:
return result
return result
return retry_agentComment · 1 year ago
It’s a pass@1 benchmark. When submitting you need to check a box that there was only 1 attempt per problem. See here for example:
https://github.com/SWE-bench/experiments/pull/219
Building multiple attempts into your agent is stretching the rules, even if technically it’s acceptable
Comment · 1 year ago
I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance.
71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running.
But the top rated submissions aren’t running production products. They generally have extensive scaffolding or harnesses that were built *specifically for SWE bench*, which kind of defeats the whole purpose of the benchmark.
Take for example Refact which is at #2 with 74.4%, they built a 2k lines of code framework around their agent specifically for SWE bench (https://github.com/smallcloudai/refact-bench/). It’s pretty elaborate, orchestrating multiple agents, with a debug agent that kicks in if the main agent fails. The debug agent analyzes the failure and gives insights to the main agent which tries again, so it’s effectively multiple attempts per problem.
If the results can be reproduced “out-of-the-box” with their coding agent like they claim, it puts it up there as one of the top 2-3 CLI agents available right now.
Comment · 1 year ago
I see it a bit differently - LLMs are an incredible innovation but it’s hard to do anything useful with them without the right wrapper.
A good wrapper has deep domain knowledge baked into it, combined with automation and expert use of the LLM.
It maybe isn’t super innovative but it’s a bit of an art form and unlocks the utility of the underlying LLM
Comment · 1 year ago
It will catch those sneaky bugs
Comment · 1 year ago
Stating that Israel doesn’t have a right to exist has been recognized to be an antisemitic statement by many prominent institutions.
It’s a radical statement that effectively denies the rights of millions of people to exist and is especially problematic given the historical context of the establishment of Israel.
The statement gets thrown around so much in certain circles that it’s gotten normalized. You’ve apparently lost sight of or never stopped to think what actually means, to the point where you’re providing it as an example of an innocent statement that got you banned for no reason. Taking this statement out of radical activist circles and into the real world won’t go well.
Take some time to educate yourself and reflect on what it actually means.