disconcision
Joined 11 years ago
Joined 11 years ago
presumably one of us is wrong.
therefore, humans don't reason.
how do you mean? given that spec, ambiguous code just won't compile. that could potentially be inefficient, but not a foot gun.
I agree that the examples listed here are relatable, and I've seen similar in my uses of various coding harnesses, including, to some degree, ones driven by opus 4.5. But my general experience with using LLMs for development over the last few years has been that:
1. Initially models could at best assemble a simple procedural or compositional sequences of commands or functions to accomplish a basic goal, perhaps meeting tests or type checking, but with no overall coherence,
2. To being able to structure small functions reasonably,
3. To being able to structure large functions reasonably,
4. To being able to structure medium-sized files reasonably,
5. To being able to structure large files, and small multi-file subsystems, somewhat reasonably.
So the idea that they are now falling down on the multi-module or multi-file or multi-microservice level is both not particularly surprising to me and also both not particularly indicative of future performance. There is a hierarchy of scales at which abstraction can be applied, and it seems plausible to me that the march of capability improvement is a continuous push upwards in the scale at which agents can reasonably abstract code.
Alternatively, there could be that there is a legitimate discontinuity here, at which anything resembling current approaches will max out, but I don't see strong evidence for it here.
It was very surprising to me that claude wasn't very good at even the latter. It took several rounds of prompt refinement, including very detailed instructions, to get even 75% non-broken links, from ~30% on first attempt. It really liked to hallucinate goodreads URLs with the book title in them but an invalid ID (the title part of URLs is ignored by goodreads).
The former was less surprising... It attempted a very rough manual strategy for generating superimposed SVG outlines on the books, which often started okay but by the right side of the the image often didn't even intersect the actual spines. I tried to verbally coax it into using a third party segmenter, but with no luck. We eventually found a binary search style iterative cropping strategy that worked well but was taking 6 minutes and nearly a couple dollars per spine so I killed that and just did the SVG outlines in figma myself.
* https://andrewblinn.com scroll down to find the bookcase and drag down on it to zoom and unlock the books