whatsgonewrongg

You’re right that some humans will, and most LLMs won’t. But humans can be just as confidently wrong. And we incentivize them to make decisions quickly, in a way that costs the company less money.

A human answering every question is also not guaranteed to give good answers; anyone that has communicated with customer service knows that. So calling it impossible may be correct, but not useful.

(We tend to have far fewer evals for such humans though.)

I’ve had Claude hallucinate options for a lot of things, most recently DOM method arguments, and Wrangler configuration. It’s like, very reasonable things you might expect to exist. But they don’t.

I must be holding it wrong. How are people working with these tools to produce quality software?