StarSkirmish: Hillclimb is a long horizon implementation of the StarCraft: Broodwar based LLM benchmark that I released last week.
Each LLM writes a Protoss bot in C++ against BWAPI, played on OpenBW. They must climb five tiers of Protoss opponents, from scripted demo bots up to top rated human written bots like Stardust and PurpleWave.
The StarSkirmish bench restricted LLMs to 1 hour of wall clock time for reasoning. With Hillclimb there is no fixed time limit.
GPT-6 Astra and Claude 5.5 Opus both benefit from substantially longer reasoning periods than 1 hour and so this format is a more realistic depiction of their abilities
Each LLM writes a Protoss bot in C++ against BWAPI, played on OpenBW. They must climb five tiers of Protoss opponents, from scripted demo bots up to top rated human written bots like Stardust and PurpleWave.
The StarSkirmish bench restricted LLMs to 1 hour of wall clock time for reasoning. With Hillclimb there is no fixed time limit.
GPT-6 Astra and Claude 5.5 Opus both benefit from substantially longer reasoning periods than 1 hour and so this format is a more realistic depiction of their abilities