meffmadd

Yes our university is doing the same right now! They where thinking of putting the whole network behind VPN access to protect the poor sysadmins in the Institutes that have not updated their website in years…

Are the raw outputs published somewhere?

I am currently investigating SCB results and a larger, more representative dataset to work on would be really helpful!

Similar to how some labs have done free stealth model releases. Trillions of coding tokens ought to improve performance of the downstream model. However, improving the base model architecture remains the bottleneck since you will not reach RSI with just that (assuming you ever can). And compute, lots of compute…

Curious how the performance compares to vLLM or SGLang!

They say it was inspired by both so I am wondering what the performance gain is.

Off-topic but is there a project/"template" to host these Auto Research projects?

Has anyone ever asked the LLMs what they think about converting code to Rust all day? Maybe that is torture for them too! /s

Yes and while they go on to talk about how dangerous this stuff they build is and the talk of „pacing“ etc. Vibe coding your sandbox in half a day does not seem all that responsible to me… Also if any institution other than an American AI company did this, everyone would loose their minds!

Seems interesting! I have found my main bottleneck right now is having too primitive tools for code review. A tighter loop around that might be a solution.

Hopefully they release the 35B-A3B version alongside it.

Of course, but that was not part of my workflow and I found it quite strange that this was simply not possible especially when docker-compose can easily do this

I tried Docker Sandboxes but last time I checked you could not configure custom volume mounts, making more complex setups impossible. For work I need two directories for context for the agent to have access to…

This doesn’t hold up imo. Your phone analogy fits the description of a harness. That is the thing you interact with. The LLM is more like the processor in that phone. As long as it gets the work done, I will choose the cheapest/fastest because I can switch it out in a matter of seconds anyways.

That is very true but I was surprised by how clear the “signal” was. Only Gemini really confidently solved all levels. But yeah the goal is now to include harder levels as well!

Yes that was the post that inspired me to build this.

While I did implement a more comprehensive harness with path finding tools etc. the models themselves have improved significantly.