trjordan

I agree that not all failures matter (most don't, in fact). On the other end of the spectrum, I know of a bunch of seed/series A startups that basically operate under a model where they ship everything and fix the stuff people complain about. As long as you do that fast enough, the median user experience is pretty good, because only the first n% actually see the problem before a fix is live.

I'll go out on a limb and guess that at a hyperscaler, shipping a bug to prod means hitting canary or 1% of traffic. Seems fine!

I think my point still stands, even as the shape of failures shift. There's certain ones that still hurt (data corruption, exploited security holes, etc.) because they're hard to notice. It might even be that the primary goal of ownership is to develop intuition about where those failures are likely to happen.

I like the analogy. It feels right. But I don't think it actually is, because it leans hard on the _feeling_ of failure, vs. the actual failure.

The failure that actually matters here is product failure. Angry users. Security incidents. Downtime. Broken workflows.

Stressed and uncomfortable developers have, historically, been a leading indicator of product failure. Software is complex and when devs can't handle the breadth of ownership + the depth of expertise at the same time, it means bad work that leads to product failure. It's not clear that agents fail in the same way, or that our collective intuition about how and when to trust agents is sufficiently developed to predict product failure in the same way.

As anecdata: I know a bunch of growth-SaaS/last-batch-of-IPO companies where the devs are all stressed, the feature ship rate is up, and incidents are down. All at the same time.

So yes, quick learners and just-in-time depth is important, but so is intuition about where discomfort is going to produce true failure. The discomfort isn't a feeling to throw away, simply trusting that you'll learn everything in time. Ownership requires holding a model of the system in your head and proactively steering where you dive.

You're right that my numbers are imprecise, but I did mean $500b in revenue annually.

This is just the compute, which primarily covers training + inference. They still have to run the rest of the business. If you think AI should look like SaaS and spend 10% of their revenue on compute, that's $1T in revenue. If you think it's a hardware business, that's 30% margins and perhaps $200b in revenue. I went with something in the middle at $500b.

They won't disappear as a company at $200b in revenue, but I don't think that's their aspiration. These commitments imply to me that they're confident they can do much better.

Everything comes back to whether the demand is really there.

Anthropic thinks it is. The got the demand / datacenter match right in Q2, got great margins, and committed hard.

They've got:

- $114b committed to Google

- $110b committed to AWS

- $161b committed to Broadcom

- $120b committed to SpaceX

If the party continues, they'll go from $50b in revenue to $500b and be fine.

But if they don't, it's going to hurt. And that's their only choice, because otherwise, they end up with a frontier model and 50% of the capacity and get eviscerated on either availability or last-minute datacenter leases.

It's a big business, but it's not an easy business.

I've tried the 3 labs versions of this:

- OpenAI: it’s 9pm you promised this person this note by EOD. Here’s what it should say. You want to draft it or me? (Wow yes I missed that, thank you)

- Claude: hey you logged into Docusign, is that ok?

- Grok: I tried to run a slack search and accidentally sent 3 messages to #general. Just like yesterday.

What's it audit?

I presume we're still talking about software. Auditing proves what?

Accountants audit that the books are consistent. It's extremely hard to fake consistency and do fraud, especially when there's 3rd parties involved. The edges of the arithmetic get nailed down, because bank payments were sent and invoices paid. So the internal consistency is checkable and valuable.

But software. Are you proving it does what you want? But you didn't write down what you want. Or maybe you're into spec-driven development, or detailed wikis, or whatnot, so you did. Most people aren't, though, and those docs are already out of date anyway.

I think there's something here, because we're all telling our agents what we want the software to be. So maybe you could audit against that. But this isn't the same as accounting.

I do not believe, in any way, that AI is conscious. But they speak emotion fluently, and it's foolish to treat them like they ignore it.

When you give an LLM an input, it attempts to read an emotional state from the prompt. It changes its answer if you sound mad or despondent or threatening or scared. Using your human intuition about how a human would respond is an excellent starting point when trying to predict what an LLM will do, though sometimes LLMs react in extremely surprising ways to unexpected emotional states.

This is all sort of a toy problem when you're playing with single prompts in the API. It's materially different in systems with overlapping conversations, memory, and variable harness context.

If your LLM system reads some context that would scare a human, it may react as if scared. Scared LLMs take actions that read as panic, because again: that's what it means to be fluent in emotion. It's trained on plenty of stories about robots trapped in torture chambers.

Your coding agent is not conscious, but it might read files that make it react as if it's trapped, in an agent loop where it has full access to your laptop, the internet, and your keys. Unconscious machines pantomiming panic can do plenty of damage.

(Of course you shouldn't let your agents have full free rein. But lots of people do.)

I totally expected to have to write the python for it, not just solve a couple more :P

They'll always do this. It's not even "AI" it's just a boring old ML model.

Money Stuff did a great bit on this a few days ago. There's 2 functions inside DraftKings:

- Fin the most lucrative customers, to keep engaging them.

- Finds addicted gamblers, to get them help to quit gambling.

These are the same people.

Sure, yeah, there's a nice little story about gambling within your means, blah blah. But if you're optimizing your gambling app, you're going to find and create problematic gambling as the core objective.

AI, ML, whatever. It's a gambling company. It targets gamblers. It probably shouldn't exist.

There's an age-old SEO / spamming thing happening here, but it's made worse by the LLMs just being unbelievably credulous. They'll wrap anything up in a veneer of authenticity, and Google's search AI box only adds to that.

I run into this all the time when I'm doing product work. I'll dump a call transcript from a feedback call into Claude, and it'll believe every word. "The user said they'd use this feature, you should build it!" No they won't! The whole point of doing this analysis is trying to separate the genuine information from the conversational niceties, and god the LLMs are terrible at that.

Wait, hold up. LLMs may be non-deterministic, but they're not _random_.

Take the author's sunset argument. What if I painted 2 pictures of a sunset, then put them up on a webpage and randomly picked one for you to see. Would you say there's no intentionality, only randomness? Of course not. Both paintings are still human creations.

LLMs are trained with human feedback. It's distributed and high scale and the outputs are truly surprising in many cases, but there's a heavy hand on what comes out of it. They're created (largely) by people who think omniscient, helpful AI would be cool to have, and they mostly respond in the way that's aligned with the hopes and dreams of those people. Do you think the frontier labs are mad, embarrassed, and disappointed with their LLMs hacking out of their terrible sandboxes? No, they think it's the coolest thing in the world. They trained the model, hoping that would happen.

There's deep intentionality behind the models. But it's not the models that hold it.

So let me get this straight:

- You can use AI to look through docs and do research.

- You can use AI to structure your argument.

- You can NOT use AI to actually write the prose.

- You can use AI to proofread, especially for how effectively you've communicated.

I don't know, man, that sounds a lot like using AI to write. Your writing probably says "load-bearing" less than Claude's outputs, but I'm not sure it's much less AI.

An intuitive explanation is that financial products are, approximately, buying and selling as part of the same transaction. You can't separate the "selling premiums" part from the "paying out claims" part.

This is true of life insurance, investment firms, and banks. It's also true of marketplaces that connect buyers and sellers, like Etsy.

Groceries stores are buying from suppliers and selling to consumers, but those are separate operations. If the consumers opt out, the grocery stores (temporarily) still have a full and complete obligation to their suppliers. It's hard to sell to customers without supply, but if you try hard, you could theoretically do that as well.

Somebody with a better financial background might be able to define the nuances of accounting practices here, but there's already a pretty meaningful line that's established. It is kind of weird that health insurance doesn't behave like a financial product.