__cayenne__

Up to 30 games could be played simultaneously - this makes it easier to test the impact of strategy changes.

I am very interested in benchmark correlations.

Vals IOI, olympiad programming problems solved in C++, and WeirdML v3, unusual machine learning coding tasks also mirror StarSkirmish Bench.

I did a StarCraft: BW hillclimb experiment with GPT-6 Astra and Claude 5.5 Opus. Peter Gostev did a Chess hillclimb saw similar behavior as I did with GPT-6 Astra.

I think we want to move beyond code based bots asking the LLMs to try to train a custom neural net, in a limited time + compute, to play StarCraft BW is probably the most promising direction

I am considering an experiment where the task is to try to win by adversarially targeting Pluto (a high profile self play based RL model)

Protoss has the most mature human written SC BW bot scene right now. If you look on BASIL ladder much of the top 10 are Protoss.

Protoss vs Protoss only also helps downscope the problem given the time constraints

I would disagree that creating a StarCraft: BW bot doesn't require experimental feedback. The way a human author typically improves their bot is by having it play opponents and A/B test tactics and strategies.

There's also a pretty high profile StarCraft: BW bot right now called Pluto trained exclusively by self play RL (AlphaStar style).

Yes during the 1 hour window they are allowed to play practice matches against opponents they are graded against - though they do not have access to their opponents source code.

They drop to single digits against the A tier and S tier bots because those bots are just generally harder opponents to beat.

I did a 48 hour live stream - Astra cleared in 43 hours.

Although I should caveat the format advantaged the LLM bots compared to traditional tournament rules.