alextheparrot

their RP1 IO chip is such a disappointment for high performance GPIO use cases as well, literally can’t trust any IO decision they make after having hacked on that chip

> Integrating an imperceptible, robust, and content-specific watermark

From the system card someone linked elsewhere in the discussion

No LLMs are calibrated?

I mean, you could also frame this as an issue the electorate could actually prioritize instead of just hoping the courts work it out

The derivative being a grad(ient) student sampling scaffolds against evals + qualitative observations: most prompt-based llm papers

Cancer is a parasitism that kills the host (or the host dies from other causes and it is not self-sufficient). Just because something is defined by uncontrolled self-replication doesn’t mean it is stable to live forever (Which is as much a comment on homeostasis as self-renewal)

It isn’t actually very wrong. Your example is tangential as graders in school have multiple roles — teaching the content and grading. That’s an implementation detail, not a counter to the premise.

I don’t think we should assume answering a test would be easy for a Scantron machine just because it is very good at grading them, either.

> Good evaluations write test sets for the discriminators to show when this is or isn’t true.

If they can’t write an evaluation for the discriminator I agree. All the input data issues you highlight also apply to generators.

I wish the other replies and this would engage with the sentence right after it indicating that you should test this premise empirically.

If this sort of error isn’t acceptable, it should be part of an evaluation set for your discriminator

Fundamentally I’m not disagreeing with the article, but also think most people who care take the above approach because if you do care you read samples, find the issues, and patch them to hill climb better

LLMs evaluating LLM outputs really isn’t that dire…

Discriminating good answers is easier than generating them. Good evaluations write test sets for the discriminators to show when this is or isn’t true. Evaluating the outputs as the user might see them are more representative than having your generator do multiple tasks (e.g. solve a math query and format the output as a multiple choice answer).

Also, human labels are good but have problems of their own, it isn’t like by using a “different intelligence architecture” we elide all the possible errors. Good instructions to the evaluation model often translate directly to better human results, showing a correlation between these two sources of sampling intelligence.