jonahx

Yes, you're right, understanding informs the constraints and the possible solution spaces. That should be added to the list.

Practically speaking, I don't think the "black box" metaphor is as broken as you're saying, though I like the point you made. There is still a ton of software (the majority, I think -- crud apps of all flavors) where an unthinking, untechnical PM wants something they can specify, and an LLM can give it to them, and it will more or less solve their problem. It won't be the best possible thing, but it will be good enough.

I don't disagree with the points you've made but what are they rebutting?

The economic value is still the product you delivered. Your understanding of the implementation meant you participated in the work of the "product manager". I think most good devs do. That might be an argument for devs to transition into designer/PM roles. That is effectively happening already, whether or not role titles have caught up.

But in any case the value isn't in the understanding itself. It just happens that the understanding still helps create better block box products. Also, I wouldn't count on this "human advantage" lasting long. I hope it does, but ime LLMs are now good enough to do this "co-design" too, and will only get better.

The parallel is real, but in math the "cash value" of the results often won't appear for years, if at all, and so the teaching and "intellectual fortification" is the primary point of the whole enterprise (at least, a major part of it).

In contrast, the value of a program is typically just the "black box" service it provides. Indeed, the program itself has always been opaque to most business stakeholders. Before, they paid human programmers and got back a program they didn't understand but could use; now, they pay an LLM provider and get back a program they don't understand but can use. In the pre-LLM era, the economic value of "understanding" and "shared knowledge" about the programs themselves was only that it made maintenance and changes cheaper. It never had intrinsic value, except as intellectual satisfaction to programmers (I was... am... one of those who found that value).

> Limited in what way? Its basic structure is sufficient to eventually let us discover anything there is to discover. It's a fully general problem solving framework.

That "eventually" is doing a lot of heavy lifting. It's limited in the most literal, empirical way. We don't have answers to most things. If we stood in relation to a gargantuan code base as we stand in relation to the natural world, with an equal proportion of known vs unknown, we'd be pretty screwed.

As an aside, even with the "eventually" qualifier, the claim isn't true. The generation of hypotheses, in particular, and even sometimes the experimental setups to test them, happens through an "out of band" process -- namely, human creativity and intuition. Which is effectively a computational process we don't understand, and limited in its own way (despite its incredible productions). The history of scientific breakthroughs is the history of astonishing insights out of nowhere, from the brains of individual people, sometimes literally from dreams. The experimental method provides a reliable way to test the insights more or less "mechanically", but it would be mostly impotent without the other part.

> we have effective tools and processes for each of these search spaces

Do we? Science (the tool for the natural world) is limited and there's so much we don't understand.

The recent superhuman achievements by llms in math are promising, but what we've seen so far isn't enough imo to YOLO building arbitrarily large systems in code and just trust that the llms will be able to handle it. Especially when the stakes are high (human lives, etc).

GP is saying that "the search space" of "the natural world" is much bigger than the search space of "stuff created on purpose by humans". In both cases, there will be guesswork, but in one case less.

I think there is something to this, but am even more concerned by the pure scale. Consider 100k, 1M, and 10M LOC human legacy projects (already a big problem, usually!), and now compare to the likely outcome of 100M, 1B, 10B, etc LOC llm projects. That's where you may start to see the outcomes I reference in my OP.

Right, that's what I alluded to in my last sentence.

It is true that this gamble has existed in the form you mention for many years. Still, I sense something new here, both in scale and in the attitude of trust. Also, I find those massive, incomprehensible human-built systems frightening too.

> have the LLM focus and summarize a slice of the system when they need to know about it or ask for it. But.... That's not knowledge? That's just search a la google? And thus seems to be subject to the same problems where the quality of your search dictates the quality of your knowledge that you surface at any given time?

This is the crux. Take pause, because hidden here is a sea-change in our fundamental trust model of software...

There is no longer anyone to turn to if this system breaks by, eg, having a critical bug that no one can figure out. No causal model in anyone's head that can be appealed to, to slowly and methodically debug. By design, the system is too large for that, and everything hinges on Claude itself being able to find and understand the right slice of the system in any situation. As long as that very new invariant holds, everything will be all right. If it fails catastrophically, there is no recovery. You are the owner of a recovered alien spacecraft with tech you have no hope of understanding.

The new trust model is: We trust Claude to always understand the system enough to debug anything or make any reasonable change we could want. The risk might be worth the reward. But it's worth explicitly laying out the deal we're making.

Tbf, it's worth pointing out that many large legacy systems already work like that (sort of), with "Claude" replaced by "whoever happens to be maintaining it this year".

We seriously looked into Alpha School, my impression was strongly "overpriced con". Frankly, I don't believe their marketing material, which is what you're quoting. I am not anti-AI in learning (LLMs have a great learning boost for me personally), but learning especially for children is a complex problem and truly making it more efficient will take more innovation than an ipad with a magic program. I would caution more skepticism with these results, especially self-reported ones.

You can honestly acknowledge what both industries do without "being out there trying to ban them." In any case, plenty of people want to improve working conditions in the agricultural industry. Probably the best you can do in both cases is imperfect regulation which at least tries to change incentives for the worst kind of abuse.

I never argued to ban pornography, and I'm not in favor of that.

That said, no, it's not like other industries. The amount of well-documented abuse is much higher. Degree matters.

This is not the claim. The claim is that, empirically, many (or enough) of the women who end up in porn are underage, trafficked, or psychologically coerced (due to poverty, drug addiction, abuse, etc) such that the choice is not really the free choice of willing participant.

I am all for individual freedoms, but I've read enough about this issue, and seen enough documentaries, that I think the claim is very well-founded. Of course, some porn stars are just willing adults doing something they want to do, but there are many who are not.

I don't understand the relevance of time to fix. It has no correlation with severity.

The headline here is that none of the bugs were serious.