Comment by ameliaquining

Hang on, can someone help me understand how these benchmarks work? The post alleges that at least some computer use benchmarks are broken because the model is making direct CDP calls instead of actually interacting with the UI. But the major frontier models' computer use interfaces don't work that way; the model only gets access to a virtual display, and can only provide virtual keyboard and mouse inputs. There are separate browser-use interfaces that offer a CDP connection, but presumably you wouldn't use those if you're specifically designing a computer use benchmark? Is the post just claiming that a lot of impressive demos that people think are of computer use are actually of browser use? I wish it'd been clearer about this.