alexchengyuli

Sadly many people wouldn't even notice 500MB anymore. That's kind of the point. A PS2 game was 4GB. Now a single update patch is 50GB. Software stopped being designed for everyone a long time ago. It's just more obvious when it's a 500MB article about RSS readers.

A 4-byte key and a 32-byte key both produce output that looks like ciphertext. Unlike most bugs, crypto bugs don't produce visible errors. That's why this one survived 12 years.

Hammerstein's matrix works because armies have rank, evaluation, and consequences. The classification is useful because there's a system behind it that can observe, judge, and act.

But if "stupid" just means can't adapt and repeats what it's told, that's every script and cron job ever deployed. We used to call that automation. Humans can be stupid too, and we don't fire all of them. We put reviews, audits, and liability around them.

AI agents have none of that. A junior dev who breaks prod gets a postmortem. An agent that modifies 50 files gets a git revert. The problem isn't stupidity. It's that there's no structure around it.

The paper puts AI next to System 1 and 2, but those are ways you think. With AI the thinking still happens, you just can't see or control it anymore.

When you googled something and got five contradictory results, that told you the question was hard. A clean AI answer doesn't give you that signal. Coherence looks the same whether the answer is right or wrong.

The failure mode didn't get worse. It got quieter.