kissgyorgy
Joined 14 years ago
Joined 14 years ago
Also there are harnesses (e.g. omp) which are using multiple LLMs debating, questioning, reviewing other LLMs output and you get a pretty good result at the end that way, which makes your job of reviewing them faster too.
The way I rather do it is tightly control the output by skills written yourself, prompts, plans, etc. and have the closest possible outcome you would write yourself.
we expect substantial benefit for legitimate defensive work, while meaningfully constraining prohibited offensive use.
That's literally impossible. Writing an exploit agains a known vulnerability needs the exact same knowledge that defending against the exploit of the same vulnerability.Also just making the model better at code is just making it better to writing offensive code.
Somebody wrote [1]; "I am never touching Minimax or GLM again. Their APIs had constant outages and I had to restart my runs multiple times — after burning money on the runs that failed midway." and I 100% agree.
The model might be good, but if the API is so bad, it's effectively useless.
[1]: https://kasra.blog/blog/i-spent-1500-seeing-if-llms-could-ha...
The biggest difference is state, but that's also kind of easy from CLI, the tool just have to store it on disk, not in process memory.
I do exactly the same, I even developed my own workflows wit Pi agent, which works really well. Here is the reason:
- Claude needs a lot more steering than other models, it's too eager to do stuff and does stupid things and write terrible code without feedback.
- Claude is very good at following the plan, you can even use a much cheaper model if you have a good plan. For example I list every single file which needs edits with a short explanation.
- At the end of the plan, I have a clear picture in my head how the feature will exactly look like and I can be pretty sure the end result will be good enough (given that the model is good at following the plan).
A lot of things don't need planning at all. Simple fixes, refactoring, simple scripts, packaging, etc. Just keep it simple.
https://kissgyorgy.me
[ my public key: https://keybase.io/kissgyorgy; my proof: https://keybase.io/kissgyorgy/sigs/_h3FGDf9mSW2s3hEXdZaErfM_wow-flxro_dDRxnnlA ]