I've wondered if these types of casual open world, resource collection games-- which are basically a series of mini-games-- would make good model training envs. Each mini-game has a pretty simple mechanic that needs to be learned, as well as the navigation and interactions to begin it. Those could be pretty useful generic skills I think.
One of my side projects is a game, and I've made it so agents can spawn off their own world shard to play in. There's a little Chrome-driven agent harness that can grab low res screenshots while it speaks directly to the game over the wire to control things.
I've had Luna play it and it can drive around (you play as a bike) but it did struggle to play one of the mini-games (Disc Golf). My idea was that model trainers could run it an RL loop using the harness to get feedback data and screenshots to use, then reset the game world as needed during the process.
A smart model capable of multi-modal inference on screenshots/game context at 30-60fps could be pretty interesting, and I'm sure it's coming. No idea how much it would generalize but would be fun to see done.
One of my side projects is a game, and I've made it so agents can spawn off their own world shard to play in. There's a little Chrome-driven agent harness that can grab low res screenshots while it speaks directly to the game over the wire to control things.
I've had Luna play it and it can drive around (you play as a bike) but it did struggle to play one of the mini-games (Disc Golf). My idea was that model trainers could run it an RL loop using the harness to get feedback data and screenshots to use, then reset the game world as needed during the process.
A smart model capable of multi-modal inference on screenshots/game context at 30-60fps could be pretty interesting, and I'm sure it's coming. No idea how much it would generalize but would be fun to see done.