ninjahawk1

Really cool idea, you might consider bundling taps when you click a video so it auto-plays instead of opening up the native web browser for youtube. That would make it feel more like the official client. Could easily hide it behind a “buffering” screen.

I’ve done my own experiments with something similar, minus the Minecraft and plus Qwen 3.5 14B agents on my PC.

When one agent tried to delete another, that agent shut down my entire PC to avoid deletion. I could probably find the blog somewhere if anyone is interested that I wrote about it. But the point is that this has been a concern for a while now.

Frankly, I’m a bit surprised it hasn’t caused any big problems.

cricket Hugging Face cricket

John Ternus is the most unlucky person ever, I almost feel bad for what tech execs are gonna have to deal with for the next like 2 years (before software doesn’t mean anything and hardware is priced out for consumers).

This is a limitation of any public benchmark. Anthropic could theoretically identify the prompts and treat them differently, and there’s no way for an external observer to prove that isn’t happening. Which is partly why we need more capable open-source models.

A few things make it harder: the panel contains questions drawn from multiple benchmarks rather than one recognizable test, the evaluation is automated and fixed ahead of time, and the raw outputs/results are public so odd behavior can be inspected.

But ultimately LiveNerf measures the behavior exposed through the API on a fixed public panel. It can’t prove what’s happening internally or guarantee the provider isn’t conditioning on the benchmark.

Longer term, I’d like to add held-out/private or periodically refreshed panels specifically to make benchmark recognition harder. I just don’t want to quietly change the current panel, because having a fixed instrument is important for the longitudinal comparison.

The 10-day window is mainly a tradeoff between sensitivity and detection speed. Shorter windows give faster results but are much noisier; longer windows give more statistical power but could take weeks to flag a change.

Also, it isn’t comparing one 10-day period once and calling it done. The window rolls forward daily, and a change has to clear the pre-registered 99% threshold in two consecutive windows before it’s flagged.

Ten days isn’t sacred, though. Once there’s enough longitudinal data, one of the things I want to evaluate is whether that window length is actually well calibrated or should be changed in a future version.

Hi, I’m the author, The Opus 5 substitution was a validation test, not the primary measurement. At that sample size the accuracy difference was -3.8 ± 6.3 points, so it did not clear the pre-registered 99% threshold. Interestingly, output tokens moved much more (-23%), which is why token usage is tracked as a secondary signal.

The actual 10-day windows contain substantially more samples than that validation, but I haven’t demonstrated that they’re sufficient to distinguish a same-family swap of that size, so I’m not claiming they are.

The goal isn’t to make the instrument say “nerfed.” A null result is a result too. I’d much rather publish “we couldn’t detect a change of this magnitude” than overclaim what the data can support.

Author here, agreed that a single generation isn’t meaningful. That’s why the benchmark is designed around distributions rather than individual outputs.

The initial calibration screened 2,336 questions with 4 samples each, then selected the 78 questions where Opus 5.5 showed useful variance. The panel is run daily, and the actual decision is based on paired per-item differences across 10-day windows with clustered standard errors, not on any single day’s result.

The n=1 in the daily sampling rate means one sample per item per day, not one sample for the experiment. By the time a window is evaluated there are hundreds of observations, and a change has to clear a pre-registered 99% interval in two consecutive windows before LiveNerf calls it a change.

The nondeterminism is basically the reason the statistical part exists in the first place.

When you simply large amounts of information, the knowledge probably shifts to upper level reasoning, then complex reasoning develops on top of it again. But who knows.

This is a fictional short story and no claim made reflects any real world event.