drooby

Let's be more ambitious with the scaffolding and tooling that makes software correct, secure, and easy to reason about.

What do you mean?

Are you saying he's not accounting for the multi-verse where other worlds killed themselves and we just so happen to survive?

I'm a staff engineer on team of 20+ engineers working in a single codebase. We're all pretty heavy users of AI, like most engineers today.

I worry about a lack of understanding creating risk, and we are creating tools and conventions to address this risk.

But I don't yet see evidence that we're losing touch with what we're building.. yes, code is shipping faster, but I'm honestly just not seeing this "sleep-walk-engineering" issue in reality.

Code isn't shipping so fast that the engineer doesn't understand the intent and general implementation details of the PR.

PRs are required to be small (400ish lines), and have tests, and good PR descriptions must be present. And two humans must approve the PR.

The time it takes to understand, verify and validate the change is fast, often as fast as or faster than the time it takes for AI to implement.

As it stands right now, our level of understanding, PLUS our new verification tools... is all we seem to need.

We're not like the hyperscalers, but we're considering how some of their approaches might fit into our workflow.

And much like Anthropic, we aren't seeing bugs increase with the rate of code increase.

Engineers are still able to understand enough of what they need to.

Well.. again, you might be in for a surprise if you read the article. The also use codeowners to require authentication related code receive human eyes.

You're in for a surprise if you read the post.

The post explicitly says they use codeowners to require a human in the loop for billing related code.

I use Claude Code every day. And I find the product to work smoothly, and effectively. From the user's perspective, that's good software. I'm impressed with what they've built. Truly. They are using a novel approach to write good software. That's fascinating. TBH this community has lost touch with reality. AI is making people lose their minds.

The US government...

Starshield is owned by the US government...

I agree. AI + Discipline is the super power of the next generation of software engineers. Crafting guardrails on one's self, and the AI.

I think OPs point though is that, like addiction, discipline is hard to maintain. Humans are irrational, and sometimes, some people just need to go cold turkey.

It's a great analogy.

In short: explore versus exploit. Both are necessary, both are valuable.

I largely agree with the OP, because I can only see an explosion in both camps.

The explore side might be automated by compute, and that breaks the social contract of the existing academic incentive structure.

But people who are inherently curious will continue to be inherently curious. Nothing will stop anyone from exploring their curiosity, and the speed and depth at which they explore will only increase. We will build tools that enhance and automate our pedagogical compression.

And for the people who want to cook to feed the family, that automation is coming too.

Or perhaps we'll find a new ceiling: the frontier failure modes that still require a human mathematician in the loop, because discovering some class of novel insight doesn't scale with current AI architecture.

I too noticed this and liked it..

This feels more like an "e-bike for the mind" and less of a chauffeur.

You might be misreading him.

He doesn't make a claim that implies "don't train" might not matter in the way you might think.

They cannot rule out that the data was trained because they have no per-user provenance tracking through the training pipeline once data is de-identified..

The entire point of de-identification is the inability to know the source of data. If the researcher forgot to hit "do not train" then that's that..

The only thing they have is a coincidence and the fact that the LLM may have used the training data that then researcher technically may have agreed to share.

Whether or not that's smoking gun of anything is hard to say. And the fact may remain that the proofs are significantly different, we do not know.

It gets an incredible amount of mileage. And I believe it is the interface that allows the model to learn and eventually bake that intelligence into the model.

Look at how Astra scored 100% on Arc-AGI-3. It was largely because of the harness.

The harness increases the chances (often to 100% chance) of a non-deterministic LLM to perform deterministic actions.

Not only that, the harness provides the feedback that becomes training data for the model. So, over time the model bakes those lessons in, and the harness becomes less necessary and the agent becomes more efficient at some tasks.

A harness will likely always be necessary, we may hit some level of complexity or some level of compute that never allows us to bake the lessons into the model, and external tools provide the model with the ability to find leverage and make up for those short comings.

I don't believe, but this feels a bit like:

"Hey... we need to improve our porn detection AI... but we want to avoid paying for all that porn... is anyone open to taking on some personal liability for a nice bonus this year"

I'm just saying... maybe...

There is not. You'll have to explain yourself.

Perhaps you're realizing why this makes no sense?