potsandpans
Joined 6 years ago
Joined 6 years ago
Maybe you are right! I'd like to hear the justifications more though.
The difference today is that the hottest technology has also become a hot button culture war item that has for the first time split the tech community in half. This in turn creates comment sections that turn into ideological flame wars.
You could find a ask hn for ever year since inception pontificating about how the site has changed and yearning for the old hackernews.
The harness I've built for it is pretty straight forward: split binary into individual functions, give the agent a bunch of small tools to diff output with underlying binary, make them wrote clean c until there's a certified match.
Rin a final oracle to make sure the agent isn't cheating.
- Qwen 3.8 27b on my 5090, barely acceptable, thinks too long, sometimes freaks out.
- deepseek flash variants, work horse. Runinfra provides around 300 tok/s. Great for tearing through moderately complex functions
- glm and kimi k.3, expensive, slow, ultimately not that much better than deepseek flash for this task. Better at doing long context research into old compiler productions
- sol 6.1, best bang for buck. Creative enough to methodically work through more complex spellings
But I found there were around 1% of functions that these models simply cannot solve. Opus and Astra take them handily.
I'm still still in the process of looking through the solution traces, but from watching the missing sauce seems to be creative reasoning.
Sol 6.1 will just happily grind on permutations of a base shape spelling that M2C outputs, where opus and Astra immediately realize that they need to unwind assumptions taken for granted (and sometimes in the process endure a worst match).
I'm hoping to be able to capture that creativity step and distill it into the harness tools.
> If WASM+WASI existed in 2008, we wouldn't have needed to created Docker. That's how important it is. Webassembly on the server is the future of computing. A standardized system interface was the missing link. Let's hope WASI is up to the task!
How are you feeling about WASI today?
As an aside, I remember when sora 2 launched and thinking, "wow I wonder if we'll ever get something close this in open weights?" The prevailing sentiment was "maybe 5 years." Yet, we have Minimax H3 and Flux a year and a half later.
So the models _can_ use the programming language.
A somewhat flippant reconstruction of this argument is something like,
Can humans code? Or do they just remember the words? I created a programming language called Brainfuck that shows they can't program.
And then reporting results that shows some humans can in fact program in Brainfuck, just kind of poorly.
It's just not very interesting.
There's probably an overlap that makes the distinction fuzzy.
The post I read is mostly an incoherent screed.
> How could I be wowed that a Millenium Prize problem has been solved using Lean verification for its proof in 2026, when that was on our minds for our 5-year plan back in 2021? How can I be impressed that OpenAI claims to have solved it in 88 hours when even I, someone barely involved, spent more hours than that working on the tech they're using?
> How much work from how many people did it really take to get that result?
I've watched this creator since, I don't know, 2010. They seem to be a very creative mathematics educator. This the point that is being insinuated here is ludicrous.
It's like saying, "how can I be impressed by copernicus' heliocentric model, when he built it on the 500 years of painstakingly collected data from the babylonians?"
It's either disingenuous, or ignorance.
> This should be embarrassing enough for OpenAI even if they didn’t have access to Buckmaster and Alpöge’s ongoing work.
More repeating of unsubstantiated accusations fanned by the ongoing culture war.
The rest... Gee whiz. It just hits all the notes. "Anyone who disagrees with me is a pedophile and has ai psychosis."
This is a diatribe from a person who makes wonderful art videos geared towards math education, against an organization solving millennium problems.
Lc like problems has imo never been about understanding candidate capabilities. It's about the humiliation ritual of the software engineer to gauge fealty of the process.