zhwu

This blog's goal is to introduce the problem around the new NVL72 system with GPU failures in production, which not all people outside of xAI are aware of. We have more blogs coming up for what we built to handle topology-aware scheduling, defragmentation and hot swapping during failures. The short version of the scheduling piece: jobs are decomposed into 16-node slices placed atomically on a single rack, with two spares per rack reserved for hot swaps.

Seems very relevant to what you are building. Quoting what Ali said: "Quick PoC?"

RL is not just an AI engineer problem but also a problem for AI infra: many components that can lands on different places with special requirements on scheduling during launching and recovering.

We put this blog to dive into the infra challenges with some good visual. Would love to hear people’s thought on it!

It seems that cursor now supports your own sandbox

The most surprising part: the agent had access to both H100s and H200s. Without being told, it noticed H200s scored better and started screening ideas on H100s, then promoting winners to H200s for validation. That strategy emerged entirely on its own.