I'm definitely willing to be corrected, but I think it just means that this improvement is close to pure noise -- not good or bad necessarily. Just not a good signal.
To me the really bad sign is that this is the benchmark that they selected to highlight. Why not one of the less-saturated benchmarks where this harness could (in theory) show meaningful improvement over the OpenCode baseline? Seems fishy to me.
To me the really bad sign is that this is the benchmark that they selected to highlight. Why not one of the less-saturated benchmarks where this harness could (in theory) show meaningful improvement over the OpenCode baseline? Seems fishy to me.