Farmadupe

I compared the costs between luna and haiku for some classification work. Luna comes out a fair amount cheaper due to more efficient tokenization and less outputs.

  Model        input tokens   output tokens   batch cost   sync cost
  Haiku 5.5    11,893,643     240,451         $0.21        $0.42
  gpt-6-luna   7,903,468      177,602         $0.16        $0.31

ah, yes that's obviously fine; I'm definitely coming fromt he perspective of "bored guy on a Saturday evening who happened to stumble across someone else's wall of charts on HN"

I'd actually recommend some excellent books on the "philosophy" of data presentation: The first that comes to mind is "the visual display of quantitative information" by Edward Tufte seems to be freely available online, and the other one on my mind is "how charts lie" by Alberto Cairo (which doesn't seem to be freely accessible)

But if it helps, just some "initial gut feel observations" from me:

* It's definitely not possible to find issue with the the _sheer amount_ of results, but there's just far too much for a human to absorb, all presented at once

* Overall text size is quite small, and difficult to read

* The page doesn't make a strong statement of _what_ is under test: the first words are: "modern-fs-benchmark Multi-device CoW filesystems under workloads classic benchmarks skip" -- which defines the webpage in terms of what it is _not_, without stating what benchmarks are actually present.

* The first line of teh page contains run statistics that probably eithre want to b at the bottom, or just don't need to be in the webpage at all: "latest run 2026-09-18 18:50:45 UTC, kernel 7.0.0-1012-azure, 593 runs recorded · 145 trend points shown"

* A significant proportion of the free text is caveats. There's nothing wrong with being transparent about limitations, but they may be a sign that there might be alternative ways to present the data, or that the data may be flawed (depending on the caveat)

* Theres several categories that I think have been invented for the purpose of collation, but I don't think are defined on the page. I think "Overall Core" and "Core I/O" aren't explained, which means by definition it's impossible for a reader to understand the score table.

* And as we're all aware right now, current Claude models are currently struggling to write coherent English. There's several incoherent sentences on the page. It's a Claude issue.

Yes exactly. The issueThe epherrality of the VMs isn't an issue, it's the _shared_ part that's the concern here. Going by the fact that the kernel is listed as "kernel 7.0.0-1012-azure" I feel like it's a fair risk that there may have been noisy neighbours.

> compare shapes and ratios, not absolute MB/s

In this case, given that the author's own disclaimer (above) already disclaims the numeric readings, I'm not sure how it's possible to make any inference on "shapes and ratios" derived from the numeric readings.

> CI runs use loop devices on shared ephemeral VMs (one VM per filesystem): compare shapes and ratios, not absolute MB/s. Each job records a host-calibration anchor — see the table.

I think if you're not using baremetal for such tests, it's likely that the results are simply not comparable at all? What if another tenant is also using the disk?

> 2G of random garbage is written directly onto one member device (behind the filesystem's back, offset 1G — python injector; uutils dd mis-seeks on dm devices), caches dropped, then a full scrub: btrfs scrub -B, zpool scrub + wait, bcachefs scrub, md/lvm sync-action 'check' (which can only COUNT mismatches — no checksums to know which copy is right).

I'm not sure that nuking 2G of the underlying block device is a recoverable error on any filesystem that I'm aware of? Can you confirm if any ofthe filesystems really came out of the other side in a usable state after scrubbing?

-----

> Trivial-op p99, idle (ms) # A trivial operation — one 4k write + fsync every 200ms (like a shell appending history or an editor updating its swap file) — run alone for 10s. p99 of the fsync completion

In fact, if it's OK for me to ask, are any of the metrics tht you used standard industry metrics? It looks like several of the tests are bypassing the kernel's page cache? -- which I worry may fall into the trap of "I modified the system to be unrepresentative of reality and then tested it".

----

> kernel 7.0.0-1012-azure

Can you confirm if you tested on a bare metal machine? were you the only tenant?

@farlight assuming that you're the creator do you think you'd be able to rework the HTML/CSS? I'm sure you've got good data but speaking on behalf of my eyeballs, the results page is... hard to read!

Thankyou for putting the last three months of my life into words <3

> and the explanations of the agent are convincing enough that surely, it knows better than you.

I feel this in my bones. I also get to watch the misalignment feedback loop close itself when the next agent sees that security rules aren't followed because of a hallucinated 20 line justification in a code comment, and then it decides that the project _is_ a demo and then confidently writes even more security holes into the codebase.

Then when you catch the issue, the agent pushes back against the fix because it would need a schema change and production DB migration.

(If it helps, I ask my own question of myself too -- I mostly don't write code by hand any more as I find that an LLM writes it faster and with less bugs -- Is that therefore proof that my time was never worth my paycheck? I hope not but at the same time I would actually be proud if I had got away with being an accidental charlatan/fraudster at my employer's expense during my entire career)

-----

Similarly, if what I said really is true, I would be implying that LLMs are charlatan/fraudster detectors (to some statistical level). And I refuse on principle to believe that that is actually the case.

hmm, assuming that this article is part written by claude and part human-written, can anyone help me find a rule of thumb for "how to know if the article is worth reading"?

Because on the one hand, the prose and the presentation is painful (narrating irrelevant points, nonlinear X-axes, ambiguous chart labels, etc etc),

But on the other hand, the result that I'm assuming the author means to communicate ("on these evals, generation quality seems fairly good") sounds worthwhile to share?

Because I really struggle with this question at the moment. Am I allowed to draw an adverse inference that "if the writeup presents irrelevant text side by side with the data, then this may be a sign that the author does not understand the task that they are attempting to write up"?

Don't know why you're getting faded out, because having worked on exactly this engine, software mitigations were put in place. That's exactly your `anomaly detection system`

It was a belt-and-braces fix, because the primary fix was of course to manufacture the engines correctly in the first place.

The "lots of temperature sensors" thing is actually easier said than done; jet engines already have lots of temperature sensors, and they will turn the engine off if a huge overtemperature is detected, but it's not a free win to add more of them. It increases the risk of a faulty sensor shutting down a perfectly good engine in flight, which from a safety-critical perspective is a borderline disaster condition all by itself.

(technically you usually cross-correlate temperature sensor readings with pressure sensor readings to prevent sensor faults turning engines off, but still, extra sensors isn't always a free win.)

Thankyou! that title was painful, I spent 20 seconds term rewriting it in my head..

* Firefox's AI Switch Is Off. Telemetry Isn't

* Firefox's AI Switch Is Off. Telemetry Isn't off [complete second clause]

* Firefox's Telemetry Isn't Off [remove negative clause]

* Firefox's Telemetry Is On [remove double negative]

----

Admittedly I didn't read past the first two paragraphs, but I assume it is a several hundred word repetition of its tautologous title.

It's a valid shift to move onto actually trying to read the article critically (which I don't mean in an insulting way -- If you assume a writing has something worthwhile to tell you, reading it critically is how you learn the worthwhile thing)

In this article, _I_ get unstuck right at the very first paragraph:

> Same driver, same track. The LLM is the star. Seventeen leading models driven round the identical 28-realworld task lap — one harness, same verbatim prompts, deterministic grading — and the results go on the board.

It jsut doesn't make much sense to me. At best, I think it can be glossed as... "I made an arbitrary benchmark which I'm not going to explain, and I plotted the results."

------

Getting my own opinions out, this is blatant slop. It claims to be "deterministic grading", but then almost the _entire_ webpage is editorialization. Examples:

* "If you only run one model, run glm-5.3"

* "opus-5 posts the best rubric on the default panel"

* "deepseek-v4-pro is nominally cheaper still at $0.0029 [...] treat it as a batch-only option."

* " It performed well on what it completed"

Yeah, I'm with you on this, I think this is just what fable/opus-5 slop looks like now...

- "A frozen verify.py accepts the claim" (what does it mean to freeze a python script?)

- "which trains the recipe eight times on fixed seeds it can't touch" (what does it mean to not be able to touch a seed)

- "One other detail is that we gave an estimation of the speedrun noise in program.md that was slightly too large. 62 out of ~100 runs measured it themselves instead of trusting our number" (What does it mean for a "run" to "distrust" a noise measurement)

- "One important disclaimer is that our benchmark has a lot of variance" (Actually this one makes sense, but congratulations for burying the lede that your entire article is bogus.)

- "Almost every model finds the same winning ideas. What separates the best traces is what an experiment leaves behind" (This implies the the graph would show every model finding a plateau in whatever metric the experiment is measuring, but I don't see every model scoring the same in the graph)