aditya-ramabadr

Yes! The models can choose their speeds and command durations. Fable for example took so long to pause and think between giving steering commands. Astra took less time but still the latency is so high that they have to pretty much drive slowly step-by-step. And definitely any human could do this way way faster, but inference speeds, latency, and intelligence get better fast enough that maybe in a year from now the models will be even more competent at this (but still probably worse than specialized models for driving).

- Aditya, Tobias, Simon

Also heard about this! But the point of our benchmark was to evaluate frontier LLMs with vision out-of-the-box, which we wouldn't expect to have been specifically trained on driving real cars. The fact that they can do anything at all (even in an open lot cone course, at low speeds) is pretty impressive. I'm sure Qwen Drive and models specifically trained for driving would do even better.

- Aditya, Tobias, Simon

Definitely the latency/speed of JEV would be great here! Unfortunately, Jev doesn't natively take in vision inputs. (So this also means demos you've seen of Jev playing games have given full structured state, which we can't do for actual driving in real time.) We tried hacky things like doing some "System One" Jev + "System Two" GPT 5.6 Luna (or other fast LLM with vision, to not bottleneck the speed) but it isn't working very well. Some open source Jev alternatives with vision exist and we might try those sometime.

- Aditya, Tobias, Simon

Great point! Yeah latency was one of the biggest issues here. To cope with that (and for safety reasons) the cars are driving at extremely low speeds. They also get timestamps with every tool call output etc so they can, in theory, "in context learn" about their own latency and choose motion durations and control how fast their iteration loop is to some extent. But yeah, this is just sort of a fun benchmark to see how good frontier LLMs are out-of-the-box at driving a real car, and probably not actually practical any time soon.

-Aditya, Tobias, Simon

True RC car is definitely the next logical step :) Yeah as long as you set a low enough speed limit, do it in an empty parking lot, have a human always ready to brake, I think it should be relatively safe haha

You could use a real car too! (As long as low speed / open lot), I just did an experiment like this last weekend, a similar stack (build off of Openpilot/Comma 4) could work with the fly model, will email you some details if you're curious

Another great thing about Groks is sometimes they just simply do what you want to do. There's no fuss about refusals or it doing random stuff you didn't ask for or it speaking in slop or anything like that. They probably should pay a bit more attention to safety/alignment as they get better but they're great models for many use cases