viksit
Joined 17 years ago
Joined 17 years ago
this exactly. most people can’t set it up that well.
Each branch in the workflow learns how to behave, not just where to route. I show how to set up a reward, plug in an optimizer, and treat agent behavior as something you can tune like a model.
- raw accuracy is now a "vanity" metric. so the benchmarks need to get more sophisticated, and i think they're going to have to be far more task specific than hotpot or hover. they've become like the mnist of multi hop.
- in my use of MIPROv2 and SIMBA, I see a fair amount of improvements for multi hop tasks (published some of these on hn before). I'm going to try GEPA and see how it performs. so I think we're at the start of what I would call "meta learning".. tuning across a huge search surface rather than tweaking one prompt. hyper param search for higher dim spaces.
- tokens burned should be a reported result
Same accuracy, 40% lower cost. Appreciate all the suggestions, this post builds on them.
2x founder. Independent AI researcher and startup advisor.
prev: led ai @dropbox, built language models @myralabs, worked on ml @bloomreach_inc