Before the edit, 93% of MATH-500 answers hit the 32k token limit, compared to 33% after.
With 10 and 25 forward passes, looping barely decreased (87% and 86% still hit the limit) and the score didn't change much. The score jump only happened at 50 forward passes when looping decreased.
Astra edited the weights to bring out a capability the model already had rather than enhancing the ability itself. Pretty impressive.
Before the edit, 93% of MATH-500 answers hit the 32k token limit, compared to 33% after.
With 10 and 25 forward passes, looping barely decreased (87% and 86% still hit the limit) and the score didn't change much. The score jump only happened at 50 forward passes when looping decreased.
Astra edited the weights to bring out a capability the model already had rather than enhancing the ability itself. Pretty impressive.