narush

I'm a co-author on a recent blog post from METR about the NanoGPT speed-run here [1]. I think it'd be of interest to anyone who enjoyed the original post. Appreciate the good beefy runs and spend here, it's a (from my experience) not super easy to do!

(Also: just to label this comment clearly: it's written hastily from a car, and based on lighter skim of the original blog post [2] than would be ideal. Please correct any mistakes or misinterpretations I have here!)

A few callouts:

1. If I understand the experiment correctly, they start the models at the original baseline. If this is true, I have some worries about contamination. Appendix C [3] has some notes on likely contamination we observed in recent models. This makes interpretation a bit harder.

2. If you look at the token scaling plots in the original post: not all models are hitting a performance plateau. This is an important point: we shouldn't treat these results as a full upper-bound on capabilities, but rather some bound on model performance @ cost (assuming good scaffolding, etc).

3. Our post is mostly about how to _interpret_ the results given here. Quoting from our post: "If we can estimate performance as a function of cost for both humans and agents, we can measure the “expenditure horizon” as the point at which those curves cross: the budget at which humans become more cost-effective than AIs. "

Feedback appreciated. I think you can see expenditure horizon as a sibling methodology (that is much less validated) to METR's time horizon work [4] - roughly, instead of baselining against the time it takes humans to complete tasks, you baseline against cost. This may be better suited to some types of problems similar to NanoGPT.

A few notes if it's helpful:

1. This post is primarily worried about ordering considerations -- I think this is a valid concern. We explicitly call this out in the paper [1] as a factor we can't rule out -- see "Bias from issue completion order (C.2.4)". We have no evidence this occurred, but we also don't have evidence it didn't.

(The SPACE [1] framework is a pretty overview of considerations here; I agree with a lot of it, although I'll note that METR [2] has different motivations for studying developer productivity than Microsoft does.)

I think this blog post is an interesting take on one specific factor that is likely contributing to slowdown. We discuss this in the paper [2] in the section "Implicit repository context (C.1.5)" -- check it out if you want to see some developer quotes about this factor.

> This is why AI coding tools, as they exist today, will generally slow someone down if they know what they are doing, and are working on a project that they understand.

I made this point in the other thread discussing the study, but in general, these results being surprising makes it easy to read the paper, find one factor that resonates, and conclude "ah, this one factor probably just explains slowdown." My guess: there is no one factor -- there's a bunch of factors that contribute to this result -- at least 5 seem likely, and at least 9 we can't rule out (see the full factors table on page 11).

> If there are no takers then I might try experimenting on myself.

This sounds super cool! I'd be very excited to see how you set this up + how it turns out... please do shoot me an email (in the paper) if you do this!

> AI slows down open source developers. Peter Naur can teach us why

Nit: I appreciate how hard it is to write short titles summarizing the paper (the graph title is the best I was able to do after a lot of trying) -- but I might have written this "Early-2025 AI slows down experienced open-source developers. Peter Naur can give us more context about one specific factor." It's admittedly less of a catchy-title, but I think getting the qualifications right are really important!

Thanks again for the sweet write-up! I'll hang around in the comments today as well.

TLDR: over the first 8 issues, developers do not appear to get majorly less slowed down.