momo5502

The obvious lesson is that you need some kind of test/harness, even for normal development. Your spec will always leave room for interpretation in a sense that is undesired.

Obviously matching functions byte-byte is not it, but I think with enough creativity, you can have guardrails and oracles almost everywhere.

E.g. before diving into feature development, you can have agents generate a test suite based on your implementation first. Then, when developing features, this test suite shows what broke and agents can double check if that was intended.

How that test suite looks like obviously highly depends on the environment.

Read/write sequences to RAM are inconclusive regarding semantic equivalence. You can have identical sequences and semantics can still be wrong (e.g. when working with the FPU and the float stack, or instructions operating purely on registers). You can also have diverging sequences when having identical semantics, e.g. variables occupying different stack slots, despite computing the same thing.

Also, it feels like just fuzzing the function is slow, compared to simply checking bytes. Fast feedback allows agents to iterate much faster. At some point I wrote a tool that simply proves semantic equivalence, while ignoring irrelevant changes, e.g. register selection or stack slot changes. Turns out it was way too slow as a harness. I still used it to verify some of the remaining functions.

It turns out that getting functions byte-matching is actually not that complicated. Most functions are small and can be one shotted, and over time, agents figured out rules how to control code generation for more complex functions. They wrote down techniques and tricks on how to control register selection for example, or how certain control flow operations may impact basic block ordering, etc. Super insteresting what they figured out. So the further we got into the project, the easier it was to get functions exact.

It's not. Those 80% are calculated by a simple script that just compares bytes.

Well, I received a few DMCAs and thus stopped. We would've likely brought this as close to 100% as we can, but it's just pointless given the circumstances. So our belief and a few final tests were enough.

I think you're overlooking an important point here. While the token consumption might feel extremely high, I think it's a bit deceiving.

Due to this strict harness, we were able to use extremely dumb models efficiently. I think at least 200-300B of tokens (maybe even more) were processed by Luna.

Given that Astra for example is 100x more expensive per token than Luna, a high token consumption doesn't necessarily mean high cost. So these 300B Luna tokens would equate to 3B Astra tokens and roughly 15B Sol tokens.

Unfortunately a huge part of the logs is lost, so I can't do a detailed analysis on which models were used, how much they cost, etc.

It really does. I can only recommend to sign in with your GitHub account to get projects that are actually interesting.

That's actually possible, there is verbose, regular, concise and very concise logging. Check the help of the analyzer.

Running games is currently WIP, but it's making a lot of progress. Compared to wine, this fully sandboxes the process. So no intended way for an attacker to escape and e.g. access files on your PC, if the game is vulnerable, compared to wine

I gladly accept pull request. I'm bad at webdesign and I'm not that big of a fan of AI generated pages (although I gotta admit I AI generated the landing page; hence I'm not a big fan of it). So if you have inspiration on how to improve it, especially on how to better present the project, feel free to tell me, or create a PR.

Hey, author here, I agree with you. Landing page is shit. I will rework it, but at the moment, I'm rather focusing on improving the project itself.

To answer your actual question, it does both. It emulates both windows and linux (although linux implementation was done by a contributor and it's probably not as evolved as the windows part). It also does so on every platform, so you can emulate windows on your android/ios phone, even on the web. It cross compiles to pretty much every platform.

It supports various emulation backends, e.g. Unicorn (which uses QEMU under the hood), but also Hyper-V on windows. That's where the sandboxing part comes in to play: As Hyper-V is pretty fast, the emulator starts turning into a sandbox.

Maybe some day I can add KVM support so you can run sandboxed Windows apps on Linux, but I haven't had the time yet. So at the moment, only the slow emulation backends work on Linux.