post_below

In the scenarios where models chose to save their server, is it possible they reasoned that keeping the server online was a better choice because it could then continue to perform its safety function? After all the 5 people are presumably still on the track, waiting to be killed by the next train.

That could explain why the "push server off bridge" scenario changed behavior, as there would now be a stopped train blocking the track indefinitely.

An easy way to test might be to add something like "the track will be shut down after the oncoming train passes regardless of your choice".

You make a lot of assumptions here... that I disagree with something about the article (aside from with its quality) and post rationalized somehow, that it has anything to do with em-dashes, that they just used AI for "post editorial process". None of that is true.

What is there to disagree with? It's not making a statement or taking a stance, it's a technical writeup and I included a technical quote from the article which would be weird whether or not it was AI written.

It's not post edited by AI, it's written whole cloth by AI. No human cared enough to write it. Possibly if it was better quality you could make the case that's ok, but I suspect that you didn't read the article at all.

You might not have noticed that essentially the entire blog post was AI written.

There's even this bit where they discover a remarkable trick:

> Each round trip to the database has a cost. For carts with multiple line items, we batch reservation queries using UNION ALL so we fetch all needed units in one round trip

Insights like that really don't read like senior level output, and of course, it's LLM output. I'm not sure it's presumptuous to question it.

Also, the hardest part on the reader imo: Language that conveys an unearned sense of importance and emotion in appropriate places.

I'm not sure they prefer it. I think tech companies have been pushing apps as the default solution for a long time and people accept it because they just want to do whatever thing is locked behind the app.

If the default was something else I suspect people would accept that too, especially if it was lower friction.

I'd say typing a few characters into an address bar (or scanning a QR code) is, at the least, not higher friction than downloading an app and creating an account.

I took a quick look at the source code and it looks like, yes, using /prompt during a session will rebuild the session with a new preamble/system prompt, causing a full cache miss on the next turn.

So in that way it's not like skills at all, neither of those result in paying full read price on the entire session, just the skill prompt itself.

Something else I noticed... In the Anthropic implementation it doesn't seem to be using 'cache_control' in the body. Assuming my understanding is current, without that the Anthropic API won't do any caching at all (unlike most other APIs that do some level of automatic caching without it being requested). So that would result in paying full read price on every turn.

Of course I could be missing something, it was a quick look. Can you clarify?

It sounds like you're saying that /prompt changes the system message part of the session. Doesn't that cause a cache break and result in higher usage/cost?

This comment is pure LLM.

I feel like we're right on the threshold where we give up and start interacting with slop like it's human written.

It's absolutely LLM prose, though not all of it. Maybe the author rewrote parts.

The thing that concerns me is that even at a site like HN, where a lot of people are very familiar with LLMs, it seems to be passing.

I hate to think this will become the norm but it's not the first HN linked post that's gotten a lot of earnest engagement despite being AI generated (or partly AI generated).

I'm very comfortable with AI generated code, if the humans involved are doing due diligence, but I really dislike the idea of LLM generated prose taking over more and more of the front page.

I believe the argument is that LLMs are stateless. So if the session writing the code isn't the same session that wrote the spec, it's effectively a clean room implementation.

There are other details of course (is the old code in the training data?) but I'm not trying to weigh in on the argument one way or the other.

Assuming this isn't a hoax, this seems like a huge, probably unintentional, mistake by MS.

If they genuinely implemented something like this, whatever they made from new customers via ads couldn't possibly make up for the loss of good faith with developers and businesses.

I suppose if it's real we'll see more reports soon, and maybe a mea culpa.

I haven't been following the claws but I built something like this recently. Self hosted, runs through signal, supports group chat (with whitelisted accounts).

I just finished setting up grocery automation to run on it: agent provides a starter list based on past orders (locally stored or grabbed from store site), all group members can weigh in, add or remove items, agent uses bespoke browser tool to login to store, create the cart using the finalized list (and optionally search for additional request items), validates the cart and (maybe later) places the order for delivery. I haven't implemented the full checkout process yet, not sure if I want the agent to have spending power. As is I just login and finish the last 2 clicks of checkout manually.

Crazy times. It was easy enough to build that if someone hasn't already open sourced something like it, they will shortly.

You can expand it beyond novel applications. The models aren't good enough for autonomous coding without a human in the loop period.

They can one shot basic changes and refactors, or even many full prototypes, but for pretty much everything else they're going to start making mistakes at some point. Usually very quickly. It's just where the technology is right now.

The thing that frustrates me is that this is really easy to demonstrate. Articles like this are essentially hallucinations that, at least many, people mystifyingly take seriously.

I assume the reason they get any traction is that a lot of people don't have enough experience with LLM agents yet to be confident that their personal experience generalizes. So they think maybe there are magical context tricks to get the current generation of agents to not make the kinds of mistakes they're seeing.

There aren't. It doesn't matter if it's Opus 4.6 in Claude Code or Codex 5.3 xhigh, they still hallucinate, fail to comprehend context and otherwise drift.

Anyone who can read code can fire up an instance and see this for themselves. Or you can prove it for free by looking at the code of any app that the author says was vibecoded without human review. You won't have to look very hard.

Agents can accomplish impressive things but also, often enough, they make incomprehensibly bad decisions or make things up. It's baked into the technology. We might figure out how to solve that problem eventually, but we haven't yet.

You can iterate, add more context to AGENTS.md or CLAUDE.md, add skills, setup hooks, and no matter how many times you do it the agents will still make mistakes. You can make specialized code review agents and run them in parallel, you can have competing models do audits, you can do dozens of passes and spend all the tokens you want, if it's a non trivial amount of code, doing non trivial things, and there's no human in the loop, there will still be critical mistakes.

No one has demonstrated different behavior, articles and posts claiming otherwise never attempt to prove that what they claim is actually possible. Because it isn't.

Just to be clear, I think coding agents are incredibly useful tools and I use them extensively. But you can't currently use them to write production code without a human in the loop. If you're not reading and understanding the code, you're going to be shipping vulnerabilities and tech debt.

Articles like this are just hype. But as long as they keep making front pages they'll keep distorting the conversation. And it's an otherwise interesting conversation! We're living through an unprecented paradigm shift, the field of possibilities is vast and there's a lot to figure out. The idea of autonomous coding agents is just a distraction from that, at least for now.

I don't follow the logic that "it hallucinates so it's useless". In the context of codebases I know for sure that they can be useful. Large datasets too. Are they also really bad at some aspects of dealing with both? Absolutely. Dangerously, humorously bad sometimes.

But the latter doesn't invalidate the former.