uberdavid

Author here. DeepSeek-V4 replaced multi-domain RL with a specialist-then-distill pipeline: train domain experts independently, merge through on-policy distillation. This post connects that production decision to three recent papers (Neural Thickets, Sparse but Critical, Apple's SSD) that together suggest pretrained LLMs already contain dense neighborhoods of task-specific experts, and post-training is just navigating to the right one.

Curious whether anyone has tried specialist-then-merge pipelines at smaller scale, or whether the anti-correlation of experts that Neural Thickets observes holds up in practice when you're fine-tuning for production use cases.

Author here. The core idea is that when you train the same model with different random seeds, both reach the same accuracy but disagree on ~10% of predictions. The reason connects three well-established results (loss landscape geometry, the lottery ticket hypothesis, and mode diversity in weight space) into a picture where the architecture and overparameterization are doing the real work. SGD is just rolling downhill to reveal whichever sparse subnetwork you happened to initialize near.

I reproduced the key findings on an RTX 3090 (ResNet20, CIFAR-10), including the cross-seed disagreement and MIMO's behavior when you try to fit multiple "tickets" into a network that's too small. Wandb logs and code are linked in the post.

Curious if anyone has seen the seed sensitivity problem bite them in production, especially on small on-device models where the landscape is more rugged and you can't afford an ensemble.

The system card directly compares to Opus 4.6 and other frontier models on the same evals. Cybench went from ~75% to 100%, Firefox exploitation from 1 bug unreliably to 4 bugs reliably. It's true there are many capable coding models out there, but the post is about why this specific cyber capability jump happened.

Author here. This is a synthesis of Karpathy's autoresearch (the experiment loop) and OpenAI's harness engineering post (the environment design) applied to ML research with 5 practical design principles.

The core idea is that after ~20 autonomous experiments the loop breaks down because the agent random-walks through changes with no research direction and fills its context with noise. The fix isn't a better model, it's a better environment. To turn hill-climbing into autonomous research you need forced hypothesis writing before code edits, rich diagnostics beyond the final metric, and periodic distillation of meta-patterns into a strategy doc.

This is me thinking through the design in public before pointing an agent at my URM-Energy project on a single 3090. The next post will be the results. I'd love to hear feedback on how you’ve successfully applied these ideas to your own work!

Thank you! The progress on research agents is exciting, but understanding what papers are reproducible on different datasets and architectures is often the bottleneck.