frankc

I see this complaint frequently about losing track of what the agents are doing, and I agree you do need to understand your system. But there seems to be this baked in assumption that if you lose track, you now need to manually wade through this massive mess to untangle it and maybe that is impossible. I don't agree.

If you don't understand the codebase, ask the agent to explain it to you. I'm not not kidding. Modern frontier models are fantastic as this - even more so than actually writing the code. It can tell you in words. It can generate architectural diagrams and sequence diagrams. It can write tests and scripts that prove it's assumptions. It can happily refactor so that the system design is aligned with your preferences.

Once you accept this, you can stop worrying so much about it and instead focusing on building the architectures and tools that lets the agents succeed better and faster - so called closed loops or agents prompting agents. Build systems that are more easily verifiable and deterministic so the agent can write very powerful property based tests. Focus more on what and why you are building, how to make sure all external properties are verifiable and leave the internals to the agents. The code is not really for us anymore.

It could easily be the other way around - religious addiction for people can't let go of the code.

I agree but I think its also more nuanced. I am also happy to use AI to operate at a higher level faster. I think that is somewhere in between. For instance, maybe I have some specific ideas and architecture I want to try for building a durable workflow engine. I dont just say 'make me a durable workout engine'. I'm very intentional about what it's doing at a system level but I am happy to cede the low level implementation details. If things work out, refactoring those details to my liking is also easy.

My main thought is would this allow me to speed up prompt process for large MoE models? That is the real bottleneck for m3ultra. The tokens per second is pretty good.

I think it's more granular than this, though. I also like to "make computer do thing" and have enjoyed using AI. But I also like building systems, optimizing systems. I find AI is a great partner in that. I can churn out prototypes more quickly, iterate on them more quickly etc. That also applies intra-system level. I might have a theory about how a different data structure or caching layer will affect application performance. It's now so much faster to test those kind of theories, and actually building good scaffolding around them to test them scientifically.

Yes, sometimes I can also ask AI to evaluate things at the system level and it often has surprisingly good insights, but that is usually a collaboration where our powers combined comes up with a better solution. I enjoy that process, too.

I do sympathize with the people "in mourning". I feel like this is really about how your identify is tied up in what you do. I have generally identified as a command line wizard. The xkcd of the guy flying in with "perl" very much speaks to me. But AI absolutely crushes at this. It's not that useful a skill anymore. Now I identify more as a local AI expert instead :D

I think we really need to have a serious think of what is "good quality" in the age of coding agents. A lot of the effort we put into maintaining quality has to do with maintainability, readability etc. But is it relevant if the code isn't for humans? What is good for a human is not what is good for an AI necessarily (not to say there is no overlap). I think there are clearly measurable things we can agree still apply around bugs, security etc, but I think there are also going to be some things we need to just let go of.

I recently built something in the same universe - using ffmpeg to receive streams from obs to capture audio and video - don't want to get into details beyond except to say it involved a fairly involved pipeline of ray actors and a significant admin interface with nicegui. I had no problem doing this with claude. You need to give it access to look up how do things, like context7. If you are doing something very specific, you need to have a session that does research to build a skill so it doesn't need to redo that research every time. And yes, you do need to tell it the architecture and be fairly detailed with something like how you want rbac.

Using these tools takes quite a bit of effort but even after doing all those steps to use the tool well, I still got this project done in a few days when it otherwise would have taken me 1-2 months and likely simply would never happened at all.

You need to be in plan mode. Not only can it not change code, its interaction with you is quite different. It will surface issues and ask you for choices.

One of the ways the chinese companies are keeping up is by training the models on the outputs of the American fronteir models. I'm not saying they don't innovate in other ways, but this is part of how they caught up quickly. However, it pretty much means they are always going to lag.

I think this is a pretty solid analogy but I look at the metaphor this way - people used to get strong naturally because they had to do physical labor. Because we invented things like the forklift we had to invent things like weightlifting to get strong instead. You can still get strong, you just need to be more deliberate about it. It doesn't mean shouldn't also use a forklift, which is its own distinct skill you also need to learn.

It's not a perfect analogy though because in this case it's more like automated driving - you should still learn to drive because the autodriver isn't perfect and you need to be ready to take the wheel, but that means deliberate, separate practice at learning to drive.

if you are doing equity statarb, its all low touch strategies, so yes the quants 'trade' in that they can write the strategies. The traders in that environment are more like support, usually.

I think this makes sense but I wonder if firecracker would work better than vagrant for this? I haven't used it before, though. I guess it might if you are trying to run gas town level orchestration.

The skills that matter most to me are the ones I create myself (with the skill creator skill) that are very specific and proprietary. For instance, a skill on how to write a service in my back-testing framework.

I do also like to make skills on things that are more niche tools, like marimo (a very nice jupyter replacement). The model probably does known some stuff about it, but not enough, and the agent could find enough online or in context7, but it will waste a lot of time and context in figuring it out every time. So instead I will have a deep thinking agent do all that research up front and build a skill for it, and I might customize it to be more specific to my environment, but it's mostly the condensed research of the agent so that I don't need to redo that every time.

I think that is both pretty true but massively underrated in how much faster you can solve the problems you know how to solve. I do also help it finds helps me more quickly learn how to solve new problems, but I must still must learn how to solve these new problems I have it solve those new problems or things go off the rails.