sheepseb

I feel like sometimes Anthropic just wants to be Google but for AI. I get the point of all this is to make the infra for agents to run on collaboratively. I agree with using AI for diverse tasks and sometimes it helps, but I feel that if we are sending "my agent" to work with "your agent" it will just create a lot of headaches for people to check

I guess it depends on how you would add it into a game, I think that would be pretty hard to not have hallucinations if it's something like an open world, but otherwise I think it's doable with enough time and effort.

No, not shadows. I use the heading as a token, in Doom the minimap has a player token and the facing (the current 3D view), I tried using only the frame but it was harder to learn only from the wall patterns. So the agent isn't magically grokking position, its using the map. As for the Pac Man, it's based on mouth and has tokens for "keep going" or N/E/S/W direction. For breakout it uses 4 ball tokens, one per diagonal and it moves one cell per step in that diagonal, and for pong the ball has 6 tokens (combiantion between [up, level, down] with [left, right], e.g up right, level down)

Yes, there were some hallucinations, e.g: in breakout hitting a corner would destroy more blocks than expected, in doom-lite it could shoot from miles away and get a kill. The fixed was add some noise to the input frame and keep the target clean.

The core of it uses rust with no_std and on the pico we use embassy-rs with no os. At the same time, the core can be build for webassembly but it doesn't use webgpu.

For the world model training I've used an RTX 4060, but for the agent playing the game that was directly on the pico using stream-ac

It's a self contained net for each game but the data used for training doesn't overlap with the validation. I think it would be easy to merge them into a single model