ianjbutler

Right. There IS a way to convince ai to write less code.. but it obviously spends 10x tokens, whether it’s LLMs as judge and automated review or just write 10 solutions and keep 1. Partly ai companies like it this way but partly it’s unintentional consequences that happen to work in their favor.

Training for good taste in architecture actually causes premature optimization at high effort allowance, but at the same time, it is combined with an inability to even recognize / exploit that later if it should ever become actually useful. Dial the effort down, and it skips the reasonable abstraction and nothing is dry, factoring is insane.

Goldilocks for simple stuff barely works with frontier models after they are out of their comfort zone for topic or line limits.. you kind of just have to decide whether it’s worth it anyway. In practice.. constant supervision for high quality code, or such high quality system design that the code quality is less of a down side.

Some misc advice. Your first task with ai is to use it to build constraints and guard rails for the ai.

That probably means a test suite that is at least twice the size of your code base. If you started there, great, but increase it to integration and benchies if you’re mostly holding units. It actually helps a bit if you can use a different language for this too but it depends on project, ymmv.

After that you have to either let go of certain style/idiom things or enforce ruthlessly with deterministic hooks everywhere.. no middle ground there and prompts will never work.

You also need to deliberately channel chain of thought with workflows rather than hoping for the best.. most of the generated product here shouldn’t be committed or preserved at all.. non-public doc flows and ticketing systems help.

Like people, an ai can easily create code that it cant actually maintain. It’s a matter of discipline now, same as before. Always using the smartest model is wrong, but double wrong on “wide” changes like a big refactor.. it leaves you no good options if you get into trouble.

> Code always was, it's supposed to be malleable. … You can always rewrite the code.

This isn’t capturing the spirit or the essence of the metaphor. chipping away at marble or whittling has this locality to it that’s keen but myopic. Molding with clay isn’t so detail oriented, you can literally close your eyes and focus on shape, same with turning on a lathe although I wouldn’t suggest closing your eyes.

there is such a thing as being precise without being detailed or local. Before LLMs you did this with test suites techniques maybe, and a few automated refactoring tools, CSTs and AST work, sometimes macros and other metaprogramming. Yeah it happened but fair to say it was rare and difficult. Everyone is/can be a sculptor now

K8s is a reconciliation engine for desired state, like many other adjacent tech in the infra space. Hating on yaml, json, missing types, or templating layers that drive the engine is kind of silly, and usually missing the point.. like hating dbs because you take issue with a sql layer or an orm layer. Disliking one or the other isn’t a deal breaker that invalidates the value proposition, so you may want to consider the other one before you retreat to flat text in csvs.

Kubernetes can be driven with tofu and hcl.. make it operator and even driven.. use any of many sdks, cdks to drive it, or add a completely different layer for faas or orchestration if that’s the relevant vocab for your problem space. It all runs on k8s.. that’s kinda the point.

> ultimately left my partner feeling unhappy and lonely.. do our best with 24 hour days

Impossible to express just how deeply frustrating it is to have the “sleep when I want you to, not when you want to” kind of fight. Over and over. For decades after being a grown ass adult.

It’s weird that it has to be explained to 24h normies but pressuring someone to alter a part of their biology like this is wrong. Gosh honey, plastic surgery might be uncomfortable in an ongoing way for you, but I’m willing to take that risk. Gee babe, I see you need a vegan/carnivore lifestyle but please change this just doesn’t work for me.

People can always make these things their red line and force the issue I guess, find out where the other person’s red line is. The weird part is that the sleep thing is generally accepted as reasonable and polite society kinda regards the other behaviors as very unacceptable

Not op but I think maybe the connection was more grammar vs your “select set of templates”.. observation being that structure beats no structure regardless of the base tech, and that completely structure-free interactions are generally not what we want anyway

> the model will perform worse

Depends on whether and how you want to rank stability in terms of better/worse. Models are diverging on this, which seems increasingly clear.. i.e. Fable isn't stable, but Opus isn't clever, and they hit different kinds of walls. So both the theory (diluting the correctness reward) and the practice (hard split on plan/implement/review work) seems to be pointing towards a strongly multi-model and highly agentic / harness-driven / complex-system kind of future instead of singleton monolithic super-smart models.

The do-everything model with solid reasoning AND solid results, and the honest/introspective helpful agent that doesn't actively resist governance may be at odds. Stable reasoning doesn't matter for pen-testing, and correct-answer with broken processes and fragile abstractions won't matter for math/science/coding.

> I think too much credit in the AI math discussion is being given to LLMs rather than to Lean,

For sure math and software people know this, as it's kind of hard to ignore. Developers who are totally disinterested in nerdy stuff like theorem proving will very quickly hit a wall without letting their models indulge in test suites (regardless of whether they ever even glance at the tests themselves). Physics nerds get it, and if they are on more on the experimental side then they are working to give it some kind of sandbox, all the other science wonks are getting it or will get it as they work with their more computationally inclined colleagues, same for general engineering.

There's a real danger here though in that there is no reason the general public will ever become very much acquainted with the verification/validation steps that all these systems actually need to be reliable. They just hear about this or that nearly impossible thing finally being accomplished as if by magic, and they won't do the work to think about what's in scope, out of scope, or how anything can be checked iteratively.

> It’s good know I’m not just crazy wanting to do part time EMT work and keep my software career.

Not OP, but there are dozens of us. If you might be interested in things like ski-patrol or guiding.. there's loads of related NOLS certificates besides WEMT / WFR for things like avalanche, cold-water rescue, high-angle rescue. It differs by location, but something like EMTB cert often requires a "intern" type of deal with ride-alongs and intubation IIRC, which may or may not be scary, but just the schedule of it can be harder for people with another career. Meanwhile your friendly local search and rescue may need help, and many of the more casual certs are very helpful, and knowing some of this stuff is just good civics.

Average people cannot tell reliably isn’t an appropriate or interesting test tho, otherwise Eliza and markov models etc. the framing that matters is explicitly adversarial. Play like your life depends on it instead of rooting for the machine, and you can’t win?

One way to play the game is causals and counterfactuals where Humans perform at like 90%+. Models can get close to that, but want some causal cot harness, and until the routing problem is completely solved, then that will necessarily degrade performance elsewhere, say in understanding jokes or poetry.

Check out cladder benches and related, lookup roughly equivalent psych research on children, etc. Even too-good performance is a signal as well!

Certainly if you think about this stuff a bit, accept the adversarial by default framing, and play to actually win.. it’s crazy that we are going around saying this is not only solved but solved 10 years ago.

Sigh, the whole "obviously the turing test is solved" meme is annoying.

Like, if we meant that it convincingly masquerades as a shitposter, ok. But everyone still bitches about AI slop, and everyone knows the writing is still bad. How does that even work if the turing test is obviously solved?

More to the point though, if you grill SOTA models on counterfactuals, causal world-models etc, you'll trip them up in a way that actually will not work on ESL students and children. Certainly there's no way to find a person that struggles with that and is also capable of cheerful fluent erudite discussion about astrophysics with perfect grammar. Yes, it's getting harder obviously.. but detecting machines with determined, focused and intelligent interrogation remains pretty easy. If nothing else, the models are cooperative where people wouldn't be and that's a signal too.

The best progress we've made is that most people do agree that this doesn't practically matter very much, i.e. we generally recognize the stakes were always overstated. But the constant vague appeals to common-sense that "of course it's a solved problem!" always feels naive or fake.

It may be surprising to you but, caring about fairness and democratizing access to helpful resources doesn’t actually require any kind of disadvantage or a complex. What’s your deal though, since you want to attack that?

naive to think people would follow this policy in the spirit in which it’s intended, but even then it clearly selects for the most social animals amongst your students instead of the most talented. And discussion alone is all it takes to describe the right approach, so if that’s the test, it’s meaningless to protect the mechanical syntax of implementation.

Clearly in this world the frats and the sorority members get the highest scores, and after that the best looking, the most amusing or charming etc.. so unless that’s the intent the whole approach is perhaps worth reconsidering.

If you want to teach to improve your understanding the normal thing is you become a TA so that access is open.. not open every assignment to ad hoc collaboration so you disadvantage half of the students on every assignment