rnd33

I don't really see how it is as different as you say. Even if a system is stochastic, it can still be controlled, it's just a different type of engineering. Much of traditional engineering works exactly like this, with tolerances and probability rather than absolute determinism.

I think we (software engineers) are suffering from partial blindness to this aspect of engineering given that software has always been so extremely deterministic. We've trained ourselves that solving a problem means composing small fully deterministic parts into deterministic systems, and the prospect of having to work more with statistical confidence and managing failure modes feels scary and wrong.

But how do you source the observations used that would be the basis of your simulation? Or are you assuming the real world won't change? At least in my mind I don't see many domains that remain static enough where continuous sampling of the real world will not be required. Go and Chess are artificially stable domains, the rules are fixed.

Any social domain where there's competition of resources (salary, funding) will definitely not be stable, it would require constant testing against the real world.

Great article. I feel most people, even great software engineers, seem to lack the intuitive understanding that building software is essentially modeling a system, and that every model faces the same limit: the number of interactions between its part grows combinatorially as the system grows. If some of those parts or interactions produce no value (bloat), then complexity is added for no gain. This is true regardless of who created the model or wrote the code.

Instead they seem to think that AI can someone work around what is essentially laws of nature.

Risk calculations aside, don't forget to take into account how this could impact soldier morale. Just knowing that if injured a giant flying drone will attempt to airlift you to safety, regardless of how hopeless the situation on the ground looks like probably is a huge boost to morale.

> If you can direct AI and produce more value per $/time spent then a non software engineer directing AI to develop software then you still can produce a lot of value.

Even if true (which I'm leaning towards that it is), when will companies realize this?

We could be in for a decade long software career recession until the effects of poorly managed and written software trickles down to lost customers and company bankruptcies. Bad decisions in large companies take a very long time to produce clearly visible effects.

> As you develop your skills with AI, you now have the ability to develop your own software products.

I think this is a dead end. We're quickly approaching a state where software is no longer a product that can be sold, but rather a tool used by someone selling a service.

Completely agree. In my experience we're being more productive than ever but our projects are drowning in on the surface completely valid but on the larger scale meaningless changes. The bugs fixed were real, the features added are in some ways useful, but they really do not move the needle in any particular direction.

Isn't this a case of missing the trees for the forest though? The human brain is not an LLM, and an LLM is not intelligent in the same way as a human brain.

However, an LLM is a prediction machine, prediction IS at the very least one (or the most fundamental) element of intelligence. The brain most surely contains at least some kind of simulacrum of a prediction machine. How that prediction machine is used or wrapped is another matter.

If I said to you: "Blue blue blue, the color of my car is red", would you have absolute confidence in your prediction that my car is red? Or would the way I phrased that sentence make you slightly uncertain, and wonder if there's some miscommunication going on here?

Intelligence is compression, compression requires subtraction, and for some reason LLMs are not good at subtracting. To create a coherent model you kinda have to subtract correlations until only the essential parts are still there.

What I don't understand is why LLMs haven't been able to do this yet, if it's the harness or some orchestration layer above the LLM that is needed. Because fundamentally if you can identify correlations then it's just another small step to prioritize and remove lower value or irrelevant correlations.

I wonder if what's needed is to introduce subtraction tokens in some sense, and in post-training reward the model on that.

Completely agree. AI is very impressive in many ways but there is something deeply wrong that is hard to put into words. The output is probable but never true, if that makes sense.

I think this is also the mechanism behind why AI generated videos and images are so captivating at first. I remember when Midjourney first launched and it was hours and hours of a brain-melting "Wooooooow". But once you get used to it and start to identify the patterns the brain quickly labels most AI-generated content as blank space.

If the image or text wasn't created by a human, then there was no intent behind the content, there is no message or novel information conveyed, and it reads as noise.

I think the last one is by far the biggest reason for why this happens way less in humans. In every conversation, almost for every sentence, people gauge the response their words have on whoever is listening. If something didn't land as expected (frown, confusion, unpredicted response) you unconsciously adapt and try something slightly different.

It's immediately obvious by the fact that you're clearly using way different ways of speaking (tone, speed, vocabulary) when you're speaking with a friend, versus your parents, your colleagues, people you don't know, children, etc.

Maybe, or maybe you are overly pattern matching on what you what to be true?

I don't think anyone would mind having a competent manager or PM in a standup, someone that is actually contributing towards finding solutions and ways to move forward.

Isn't this only a transitionary problem though? Right now, during the transition there is no incentive for humans to write anything, it will get drowned in AI slop.

As time goes on, more and more people will recognize this problem and we'll develop new ways of measuring information quality and trustworthiness. Nothing about this problem is fundamental, it's just that we're in the middle of a very chaotic transition.

The end goal though (in my understanding) has never been for an LLM to regurgitate knowledge it ingested during its training. The end goal is to use the patterns and correlations found in internet data to generate an emergent prediction and problem-solving machine.

Whether that's possible is something we'll discover, but no one is throwing billions on AI companies for the hope of them building a giant natural language queryable internet information repository.

For well-scoped tasks I wouldn't say the code is bad, not brilliant for sure, but definitively good enough.

For a lot of problems "quick and good enough" is all that is required. I've used it a lot for managing my Home Assistant setup. Has saved me countless of hours.

In principle I could've done it myself, but I never would have, the time investment required to learn it wouldn't have been worth the value I get from it.

Subsidized from a capex perspective maybe, but the profit margins for API-priced inference tokens are likely very high.

Also, a lot of people get stuck on the idea that the main use case of LLMs is to generate more production code. I don't see it like that at all anymore. It's just as useful or even more useful as a problem-solving partner, generating throw-away prototypes, conducting experiments, verifying assumptions against code and documentation, performing one-off UI or smoke testing.