I'm using fable and opus about 25%75/% respectively. I have a 3 step process - discuss a topic and create a review, convert the review to a task file and update the design docs, then finally implement the task file in code. Normally around 300-1000 loc implemented for a change, more for a new UI page or whatever.
And honestly it's incredibly good.
- 50% of the time it'll generate code that I accept wholesale.
- 25%: it maybe makes a choice I don't like, or the implementation isn't the approach I liked, so I tweak it.
-10%: it needlessly creates semi-duplicate function logic, so it's a lot more code and execution paths become hard to conceptualise
-10%: it mirrors business logic it shouldn't, and subtly changes it. Which needs a huge amount of intervention to fix.
- 5% it goes completely off the rails. Eg decided that business/application logic should be in the database.
Prompt and check this honestly works very well (still waiting on tests with more than a handful of paying customers however!) but a FULLY automated agent pipeline? I'm not convinced. That last 15% of cases will make a degenerate codebase real fast.
Also without a CC subscription, paying per token, it would be horrifically expensive for my approach.
And honestly it's incredibly good. - 50% of the time it'll generate code that I accept wholesale.
- 25%: it maybe makes a choice I don't like, or the implementation isn't the approach I liked, so I tweak it.
-10%: it needlessly creates semi-duplicate function logic, so it's a lot more code and execution paths become hard to conceptualise
-10%: it mirrors business logic it shouldn't, and subtly changes it. Which needs a huge amount of intervention to fix.
- 5% it goes completely off the rails. Eg decided that business/application logic should be in the database.
Prompt and check this honestly works very well (still waiting on tests with more than a handful of paying customers however!) but a FULLY automated agent pipeline? I'm not convinced. That last 15% of cases will make a degenerate codebase real fast.
Also without a CC subscription, paying per token, it would be horrifically expensive for my approach.