sgillen

Yes the latest models are pretty good at orchestration, if you instead open 9 chats and let them loose on your code base on different tasks without telling them about eachother you get a mess. If you tell one agent to spawn subagents as needed to do the implementation you get much better results.

These are not run of the mill results from AI, those are surely being posted to arxiv constantly by normal researchers without any fanfare.

These are really big results from AI, if a human had come up with them we would also be making a really big deal out of them too (maybe not as big of a deal, part of the excitement is that this was from AI and more results of this magnitude may soon follow).

1TB still smells like a bug

Hmmm but in the case of math, while some of it is "just puzzles" there often turns out to be practical applications, even if they are not obvious at first. Number theory was considered the epitome of pure math with no practical applications for centuries, now our modern society is built on it (public key crypto).

I think you've just lowered the risk of a bad hire for the company, which might allow them to "take a chance" on candidates they might otherwise pass on.

I use it all the time now, switching between claude code, codex, and cursor. I prefer CC and codex for now but everyone is copying everyone else's homework.

I do a lot of green field research adjacent work, or work directly with messy code from our researchers. It's been excellent at building small tools from scratch, and for essentially brute forcing undocumented code. I can give it a prompt like "Here is this code we got from research, the docs are 3 months out of date and don't work, keep trying things until you manage to get $THING running".

Even for more production and engineering related tasks I'm finding it speeds up velocity. But my engineering is still closer to greenfield than a lot of people here.

I do however feel less connected to the code, even when reviewing thoroughly, I feel like I internalize things at a high level, rather than knowing every implementation detail off the dome.

The other downside is I get bigger and more frequent code review requests from colleagues. No on is just handing me straight up slop (yet...)

To be fair to the agent...

I think there is some behind the scenes prompting from claude code (or open code, whichever is being used here) for plan vs build mode, you can even see the agent reference that in its thought trace. Basically I think the system is saying "if in plan mode, continue planning and asking questions, when in build mode, start implementing the plan" and it looks to me(?) like the user switched from plan to build mode and then sent "no".

From our perspective it's very funny, from the agents perspective maybe it's confusing. To me this seems more like a harness problem than a model problem.

I think there is some behind the scenes prompting from claude code for plan vs build mode, you can even see the agent reference that in it's thought trace. Basically I think the system is saying "if in plan mode, continue planning and asking questions, when in build mode, start implementing the plan" and it looks to me(?) like the user switched from plan to build mode and then sent "no".

From our perspective it's very funny, from the agents perspective maybe very confusing.

In my experience a lot of tech companies, at least in the Bay Area, have all copied this system.

Not sure if I agree with the christian references being incidental ... the first book is literally a retelling of the The Canterbury Tales, all the characters are on a pilgrimage. there are a bunch of religious groups with at least one being central to the story, there are cross shaped parasites that grant eternal life.

I still think you can enjoy it without caring much about religion.

This is very interesting because I see a lot of AI detractors point to the original study as proof that AI is overhyped and nothing to worry about. In this new study the findings are essentially reversed (20% slowdown to 20% speedup).

The study was designed to have devs who are comfortable with AI perform 50% of tasks with AI and 50% without. So the problem is the population of "Developers who use AI regularly but are willing to do tasks without AI" is shrinking.

>> Are they worried that by splitting devs into groups of AI experience they might be measuring some confounder that causes people to choose AI / not AI in their careers?

The developer sample size was small (16 people in the original study) and the task sample size is larger (~250 tasks). I think the worry is variance in developer productivity would totally wash out any signal.