donadigo

Just wanted to provide some perspective here on how many things those projects need to take care of in order to get some training setup going.

I'm the developer behind TMInterface [1] mentioned in this post, which is a TAS tool for the older TrackMania game (Nations Forever). For Linesight (last project in this post), I recently ended up working with its developers to provide them the APIs they need to access from the game. There's a lot of things RL projects usually want to do: speed up the game (one of the most important), deterministically control the vehicle, get simulation information, navigate menus, skip cut scenes, make save states, capture screenshots etc. Having each of those things implemented natively greatly impacts the stability and performance of training/inference in a RL agent, e.g. for the latest version the project uses a direct capture of the surface that's rendered to the game window, instead of using an external Python library (DxCam). This is faster, doesn't require any additional setup and also allows for training even if the game window is completely occluded by other windows.

There are also many other smaller annoying things: many games throttle FPS if the window is unfocused which is also the case here, and the tool patches out this behaviour for the project, and there's a lot more things like this. The newest release of Linesight V3 [2] can reliably approach world records and it's being trained & experimented with by quite a few people. The developers made it easy to setup and documented a lot of the process [3].

The difference between this and e.g profilers like Tracy is that there's no need to insert the instrumentation manually, and you can explore the codebase much more freely to see what's going on. This is done through dynamic function relocation and inserting an rdtsc call after each line to collect the CPU timestamps. There's some more supporting code of course, but when it comes to overhead we get about ~15 instructions per line (more if it supported multiple threads).

On top of all that, the extension works really hard not to disrupt any standard Visual Studio workflow like setting breakpoints or viewing the call stack, which has been a very hard thing to achieve: the VS debugger expects that all code is on the same place as it was and that it maps perfectly to the corresponding debug information (which is not at all the case when the extension is running!).

The feature works by setting the cursor inside a function, and the extension will start showing you the code paths that are executed and what local variables are changing in it. You can also use it as a function recorder with no breakpoints - just set the cursor in the function, execute an action that calls the function and you'll be able to trace everything that the function did.

In my case, I went to implement this for Windows, but bit more advanced with inline scripting support. Basically, you insert a snippet anywhere in the code that can call various functions and one of them is `view()` which will show all locals (or a chosen object out of scope) in a Visual Studio window (it's a VS extension [0], website [1]). Calling it again will update the view with the new state. The scripting language underneath is Angelscript and it understands symbols inside your entire project (e.g you could do `view(Namespace::g_variable)` or even call functions from the program etc. The snippet hook is then JIT'ed into the target function by relocating it using info from the PDB. This took me ~1y to implement, and it's still a research phase project.