COAGULOPATH

An interesting paper from the GPT 3.5 era is "AI-generated poetry is indistinguishable from human-written poetry and is rated more favorably" by Brian Porter & Edouard Machery (2024).

Reinforcement-learning on human preferences does work, to an extent. You get LLM output that is (shallowly) highly appealing to humans. That's certainly not in doubt.

The problem is that writing has purposes aside from making hordes of (indiscriminating, low-taste) humans smile. And these purposes are actively corroded by RL'ing on human preferences.

Generally, AI generated fiction has problems similar to the award-winnning stories listed here - they are full of incoherent, "poetic" metaphors, vague gestures to narrative depth that is never delivered on, and relentless pandering to the cheap seats. They're very much "playing the hits" - saying unchallenging, nice things that no reader would ever disagree on. We discover that racism is wrong, that family is what matters, that you have to remember who you are.

They all feature simple (almost generic) motifs, such as trees that represent life. (The repetitiveness of these motifs becomes apparent when you start reading a bunch of stories. In "The Serpent in the Grove" by Jamir Nazir we get "‘People does change,’ he says. ‘But grove does remember.’". In Back and Forth by Kavyta Kay we get "Nani smiled, sad and certain. 'Trees remember everything, beti. They live long.")

It also might be a window into how literary contests are judged.

The recent crop of prize winners have striking similarities. Largely African-American or Indian authors. Stories involving "big" humanistic issues like the legacy of colonialism/slavery, the importance of remembering one's roots, gender-based violence, and so on, with the implication that the writer is drawing on their lived experience somewhat.

I strongly suspect that the prizes are not solely being awarded based on literary merit. Judges of these contests fundamentally want to feel like their work matters: that they're uplifting worthy voices. You can deliberately write in a way that flatters this notion (or get ChatGPT to write for you.)

>They needed in huggingface and they got in in days.

It's worth noting that they did not need Huggingface for anything - they had already forged flags for their tasks, and were trying to figure out how not to get caught by the grader.

Hacking Huggingface got them caught and arguably only misled them further (since OA's implementation of the ExploitGym environment was nonstandard, and different to whatever they found on HF.)

A better approach (from their perspective) would have been to compromise OA infrastructure itself (which a later agent swarm was able to do, apparently).

This AI generated post (100% on Pangram) is pretty out of date.

>On SimpleQA, a benchmark of factual recall with no tools allowed, the current leader is Gemini 2.5 Pro at 53%, so the best recall money can buy still misses half the questions.

SimpleQA hasn't been updated in a long time. Gemini 2.5 Pro is a sixteen-month-old model, not "the best recall money can buy".

>The part I find most promising is what this does to hallucination. When a fact lives in weights, a wrong fact is unfindable and unfixable.

This seems confused. LLM hallucinations don't come from the weights containing "wrong facts", they are artifacts that appear at runtime.

>When the fact lives outside the model, a wrong answer has an address. The model cites a document, so you can open the document. If the document is wrong, you edit the document

You can make any modern LLM explain its reasoning and find sources for its claims. None of this has anything to do with facts needing to exist in weights or in harnesses.

The internet is full of wrong information and I cannot magically edit it to make it all correct, so this doesn't help me.

>if a model is factually wrong a claim with a source is checkable and a claim from weights isn't.

Why? If a model's weights claim that Bart Simpson became President in 2020, why does this fact suddenly become uncheckable?

>I'd like to know a lot more about how that works.

My guess is that it works like Gemini's SynthID: by altering the logprobs of the next token.

Like, for every 10th token, instead of outputting the most probable, it outputs the 17th most probable, or something. (Obviously it's way more complicated but I think conceptually this is how it works.) No human will notice this, but a classifier trained on Claude's output will.

So it's not like the watermark is the words "le epic bacon" and Claude will output "le epic bacon" in everything. That would be extremely annoying (and easy to defeat).

I used to make those. There was a lot of creative stuff people discovered.

For example, the game's terrain was baked at launch, so you couldn't turn land into water. But someone noticed that bridges seemed to spawn water textures under them (a cosmetic detail), and if you created a Bridge1 object and deleted it in the same trigger, it would vanish too quickly for the player to see, creating the appearance that water had appeared out of nowhere.

Someone used it to make an "ice-breaker" ship, which would turn ice to water (I think this required dozens of triggers for each point on the ice.) Some of these things had thousands of triggers, which the editor absolutely wasn't designed for. (It had no copy+paste function, and the window of viewable triggers was tiny.)

Fun times.

Please do not post AI generated comments.

In the system card they seem to dismiss this. Quotes;

> (...) Claude Mythos Preview’s gains (relative to previous models) are above the previous trend we’ve observed, but we have determined that these gains are specifically attributable to factors other than AI-accelerated R&D,

> (The main reason we have determined that Claude Mythos Preview does not cross the threshold in question is that we have been using it extensively in the course of our day-to-day work and exploring where it can automate such work, and it does not seem close to being able to substitute for Research Scientists and Research Engineers—especially relatively senior ones.

> Early claims of large AI-attributable wins have not held up. In the initial weeks of internal use, several specific claims were made that Claude Mythos Preview had independently delivered a major research contribution. When we followed up on each claim, it appeared that the contribution was real, but smaller or differently shaped than initially understood (though our focus on positive claims provides some selection bias). In some cases what looked like autonomous discovery was, on inspection, reliable execution of a human-specified approach. In others, the attribution blurred once the full timeline was accounted for.

Anthropic is making significant progress at the moment. I think this is mostly explained by the fact that a massive reservoir of compute became available to them in mid/late 2025 (the Project Rainier cluster, with 1 million Trainium2 chips).

Yes, I find LLM-written posts valueless because I can already talk to a LLM any time I want (and get the same info). It's not these commenters are the Queen of Sheba bearing a priceless gift of LLM slop. That stuff's pretty cheap.

Copy+pasted LLM output is actually far worse than prompting an LLM myself, because it hides an important detail: the prompt. Maybe the prompter asked their question wrong, or is trolling ("only output wrong answers!"). I don't know how the blob of text they placed on my screen was generated, and have to take them at their word.

>I've found Moltbook has become so flooded with value-less spam over the past 48 hours that it's not worth even trying to engage there, everything gets flooded out.

When I filtered for "new", about 75% of the posts are blatant crypto spam. Seemingly nobody put any thought into stopping it.

Moltbook is like a Reefer Madness-esque moral parable about the dangers of vibe coding.

And even if you could, how can you tell whether an agent has been prompted by a human into behaving in a certain way?

Is it a success? What would that mean, for a social media site that isn't meant for humans?

The site has 1.5 million agents but only 17,000 human "owners" (per Wiz's analysis of the leak).

It's going viral because a some high-profile tastemakers (Scott Alexander and Andrej Karpathy) have discussed/Tweeted about it, and a few other unscrupulous people are sharing alarming-looking things out of context and doing numbers.

If the site is exposing the PII of users, then that's potentially a serious legal issue. I don't think he can dismiss it by calling it a joke (if he is).

OT: I wonder if "vibe coding" is taking programming into a culture of toxic disposability where things don't get fixed because nobody feels any pride or has any sense of ownership in the things they create. The relationship between a programmer and their code should not be "I don't even care if it works, AI wrote it".

Thanks, I didn't realize the situation was so dire.

> In 1920, there were 25 million horses in the United States, 25 million horses totally ambivalent to two hundred years of progress in mechanical engines.

But would you rather be a horse in 1920 or 2020? Wouldn't you rather have modern medicine, better animal welfare laws, less exposure to accidents, and so on?

The only way horses conceivably have it worse is that there are fewer of them (a kind of "repugnant conclusion")...but what does that matter to an individual horse? No human regards it as a tragedy that there are only 9 billion of us instead of 90 billion. We care more about the welfare of the 9 billion.

And they do hacky things like space elements vertically using <br> tags.