peterbell_nyc

Ahhh, I retract my previous statement. I assumed it was "do you want to accept the offer". If it is "do you want to be considered for the offer" that's a little more complicated...

Not a problem. While it is reasonable to consider the signaling impact of all your comms at work, if you take this, they'll smile at getting closer to quota and wish you well.

Reasons to accept. Reasons to decline, but that isn't one of them.

From a pragmatic perspective, I'm with Djikstra for a very specific reason.

Clearly it is possible to "program" in natural language. I have one- and few-shotted a number of apps that meet my requirements.

Generally as the number and complexity of requirements grows it becomes harder to parse them if they are in a poorly structured natural language format.

I can't think of a time that I have provided specifications in natural language with sufficient rigor that if I was to run the prompt ten times, the ten generated programs would look identical or provide exactly the same outputs for the same inputs under all situations.

Whenever I find that I need the rigor such that the code could be considered to be a cache, not an authoritative artifact, I find domain specific languages to help me to ensure the comprehensiveness of my requirements around the dimensions that I care about.

I also find such structure to aide collaboration with other humans trying to understand or extend such specifications.

So I think for code to become ephemeral, for substantial pieces of software that require the documentation of many complicated requirements, writing such specifications in natural language is not the best interface and would require substantial support from LLMs to project them in ways (flow charts, UX demos, etc) that humans could reliably understand and mutate them without introducing regressions.

But we'll see. Last year I still thought git was an appropriate version control system to use!

I love great software and agree that great software is evolved - not built. The whole point of building v0.1 is to figure out what's wrong and should be fixed in v0.2

At the same time, there are broadly three motions in the loop: - thinking/discussing (what should it do) - building (Make it do that) - using (Seeing whether that is actually what it SHOULD do)

And then of course you repeat until you run out of time, money, patience, volition, etc. For some software there is a terminal state - it truly does exactly what it should. For most you're always reaching for it.

LLMs definitely accelerate 2 and potentially can help accelerate 1 and 3. As such cycle time can be reduced. It still may take 100 turns to get what you want, but I'd be surprised if the clock time for the 100 turns would be unchanged using AI.

You HAVE to have a set of personal evals for each class of task you want to use models against at scale so you can test plausible candidates and compare output on your work against your evals.

There is way too much subtlety in what does and doesn't work for a given problem, context/prompt, tool set and eval. I can tell you Fable is generally better than Haiku, but comparing similar tiers really does depend on your exact context.

This is the distribution of usage. Spin up a bunch of loops or fire a semi-autonomous factory at a project and it's pretty easy to blow through a 20x account in a few hours if you can afford the sandboxes, CI and other infra required.

If you're running 2-3 parallel agent session with a few sub agents and waiting for you to prompt them, you'll have a very different experience!

Wait, did you let the same agent write the tests that wrote the implementation? Or was it that the plan was the wrong way round, and both the generator of code and the generator of the tests followed that same incorrect plan?

Also, what were your mechanisms for reviewing the plan against the described business outcome? Anything else you could have done there to catch the backwardness?

And finally, did you run any of it across models from different foundation labs? I'll often run important stuff generated by codex across Anthropic, grok, and Gemini with an opus or fable judge...

Wow - so one really good researchers salary/stock grant. I feel like they're underinvesting...

Yeah, this has been my biggest contention with the models everywhere paradigm. I will definitely use a supervisor pattern and advisor pattern for simpler models when I just want to throw something together interactively with Fable and a few sub-agents. But for anything I am doing repeatedly, I built a deterministic orchestrator to run the steps and I have a hard rule that the skills that run from my repo can only ever be a thin shims calling to my central system so that rather than having agents enrich a person or call an API or do research they fire off predefined multi-step deterministic plays that just might have models for classification generation summarization and/or review.

In addition to the reliability and cost benefits you also then get shared capacity broker capabilities so if you're only able to enrich so many people or only capable of doing so many CI runs if that's all done deterministically you can have some intelligent orchestration in your main server to manage that limited capacity across all your agents that are trying to use it at the same time.

LOL - too competent for their own good :(

Also have a workspace as a personal email and ended up getting a personal gmail just to try out the subscriptions before I gave up.

I have multiple anthropic and OpenAI max plans. For Gemini I just use my Cursor $200 a month plan (which also gives me the ability to try grok, conductor, etc)

I would take the other side of this bet. While I agree that the impact of any given advance is likely to resemble a sigmoid curve, I think there is a material chance of "stacking sigmoids" creating something that looks exponential.

To take a simple example, look at the progress of technology over the last ~500 years - it seems to me that the rate of change continues to accelerate despite many of the logistic curves flattening along the way.

There are still huge unanswered questions about whether or not the stacking sigmoids will favor the incumbents. But I would not definitively bet against the people with the most compute data, talent and money.

So many thoughts

AI can materially improve PRs - you just have to do it right.

Firstly there are not going to be a material number of non-AI patches written in the future. I know some people still ride horses, but when compared to the 17th century the percentage of travel primarily enabled by literal horse power is way down. Same way goes "artisinal, hand crafted code"

Secondly the problem isn't the AI - it's the crappy prompts and lack of intermediate artifacts and quality gates (deterministic and adversarial) in the generation pipelines.

Thirdly, don't disallow AI (I mean, feel free, you're doing something without charging for it - do you) - disallow crappy PRs, verbose descriptions. and bloated code that doesn't fit your coding standards.

Fourthly, ship your standards. Document what a good PR looks like, the comments and examples, the search for open PRs to ensure it isn't a dupe or a won't fix. Then ship your coding standards - have some LLMs infer them from your current code base and review (bonus - anything it infers correctly that you don't agree with, get it to re-code to your new standards).

Then set up a PR reviewer - start with deternministic gates for the format of the PR and some broad heuristics, then run it through adversarial reviews. Anything that looks great, feel free to run by a human either before the merge or after just to keep an eye on things.

The deterministic gates will cheaply get rid of 95% of the slop (AI and human - I've seem plenty of human slop over the years) so you can focus tokens on the good stuff.

The problem isn't the AI - it's the assessments. It used to be that shipping a large, well-researched essay with multiple citations was proof of work. Now it's proof of prompting, not learning or effort.

If you want to test for memorization, you need a live test to do so.

If you want to test for comprehension and understanding, require sessions with a tutoring AI and grade the level of comprehension exhibited.

And then please, please test for prompting with challenging assignments that would usually be beyond a student's skills and make sure they know how to drive the machines to do valued work.

I totally agree that there is utility in memorization and comprehension. I also know that by the time students graduate, AI will be the job and if they can't pair successfully with agentic workflows they will not be much use in the work force for many roles.

I auto tune my prompts to a locked model version based on production data used as evals with holdback data. I think the use case for this would be one off interactive prompts? For now I just run those all against an Opus 4.8 MAX and I'm sure I could downtune, although for interactive my opening prompt isn't always reflective of my overall goals for the multi turn session.

I'm just trying to figure out why on the fly routing would beat testing and tuning and locking models and versions for each class of call, with evals and auto tunes running to explore more possible models for commonly run classes of prompt over time . . .