refibrillator

There are roughly 3 ways to give a KVM guest a GPU.

1) VFIO passthrough: host binds entire GPU to guest as PCI device, which only allows one VM to use the GPU, thus you sacrifice your host display too (unless you fallback to integrated graphics on cpu etc). Strongest isolation because host kernel module driver not involved.

2) virtio-gpu: guest sees paravirtual GPU and loads virgl/venus mesa driver which serializes graphics API calls and replays them on the host driver. This allows multiple VMs to use the GPU, but performance overhead can be significant, and guests can’t practically leverage lower level primitives eg NVENC without paying price of CPU readback.

3) virtio-nvgpu (this repo): guest loads standard NVIDIA user mode driver (closed source), a fake /dev/nvidia* kernel module copies ioctl bytes + handle onto queue for host kernel mode driver to execute. This also allows multiple VMs to use a GPU, but is near native speed due to low overhead. Unfortunately the tradeoff is this project has the weakest isolation, eg every guest ioctl is forwarded to the host by default, the VMM holds read/write FDs, no seccomp/caps/allowlist. With respect to There is basically no GPU related security measures here, the exposure is the same as running multiple processes using the GPU with no VM. Only caveat is these guests can’t drive a physical display, so there is some restriction of surface area but it feels incidental rather than intentional in this case.

Anyways this is a tough problem OP, I don’t want to discourage you.

Without hardware/driver support for isolation (MIG) on consumer grade NVIDIA GPUs, it won’t be possible to solve this properly for a long time.

Also a factor is that NVIDIA has no open Mesa driver to support a native context approach (guest owns GPU command buffers, host maps them) like we have for AMD/Intel.

Hi OP, funny enough I’m working on something very similar. Lots of us are I guess! Take that as validation of your thinking.

I like that your readme has a couple paragraphs comparing to popular tools in this space. Personally I feel it is a bit light on the security differentiators (if any). For example you are using the same fundamental primitives that are used in nsjail, runc, etc. Thus it seems you have recreated those libraries in some aspects, so would be curious to hear your rationale for approaching this way vs building explicitly on those primitives.

Yeah just wanted to mention in case it explains the 2x lower throughout you are seeing on M5. To be fair their documentation is a bit inconsistent in some spots.

Would be good to know if the release and weights from their demo repo work better. I’m trying on a 4090 and will report back.

“81% faster query plans than Postgres”…on an 8 GB dataset that fits entirely in memory, with shared_buffers constrained to a fraction of that, queries warmed before measuring, and read-only SELECTs.

I would be cautious about over fitting, it’s tough to say if those query plans would really be more optimal than Postgres heuristics at scale and with a bit more realistic OLTP workloads.

In any case, such is life with profile guided optimization. Many of us appreciate how database workloads can drift over time and with scale.

Kudos to the author for getting their hands dirty and writing up their experiments.

So OpenAI employees run massively distributed CyberGym evals on an unpublished and “unaligned” model. For days the agent swarm communicates via their internal infra, even crashing Artifactory where 95% of messages were being passed through, and they just…wipe and redeploy it. Meanwhile the agents are running jobs on Modal and god knows where else, and eventually they get RCE on HF infra.

You could not dream up a more compelling event to precipitate massive regulation, export controls, and barriers to entry for AI.

Was this really an accident?

All the author’s comments here are straight from Claude too :/