martinrolph

Think of it less like a test suite and more like an exam. If you're trying to differentiate between the performance of different people/systems/models, you need to calibrate the difficulty accordingly.

When designing a benchmark, a pass rate of roughly 50% is useful because it gives you the most information about the relative performance of different models. If the pass rate is 90%+ too often, that means the test is too easy: you're wasting questions asking the model to do things we already know it can do, and getting no extra information. And if it's too low then you're wasting questions at the other end, trying to make it do impossible tasks.

I wrote a stream of consciousness that turned into a piece of flash fiction. A peek just a little way into the future, inspired by the vertigo I felt on the announcements of Vibes and Sora 2, the AI-only short form video feeds. What happens, I wonder, when the AIs that are optimising your For You page for addiction achieve their goals? What is the endgame for addictive content?

In an IoT deployment, an issue that is detected on one device (e.g. an exception on the server) may actually originate on an entirely different device (e.g. the end device sends some incorrect data). Overlock collects together the state of all the devices involved in a given request and the associated logs in one place so that you can track the error back to where it originated.

Oops! Sorry, changed the title.