namanbnsl

I think one fundamental for getting agents to work well is giving them a number to hill-climb.

If you can create an eval for something in your project, the agent can keep trying different approaches until the score goes up.

I believe building good evals will matter more than ever.

Mainly due to the looping.

Before the edit, 93% of MATH-500 answers hit the 32k token limit, compared to 33% after.

With 10 and 25 forward passes, looping barely decreased (87% and 86% still hit the limit) and the score didn't change much. The score jump only happened at 50 forward passes when looping decreased.

Astra edited the weights to bring out a capability the model already had rather than enhancing the ability itself. Pretty impressive.