Edit distance seems like a strange way to test if the model understands arithmetic ([1], Figure 3). I think `1+3=3` would be equally as correct as `1+1=9`?
Why not consider how far off the model is `abs(actual-expected)`? I wonder if there is an inflection point with that metric.
Edit distance seems like a strange way to test if the model understands arithmetic ([1], Figure 3). I think `1+3=3` would be equally as correct as `1+1=9`?
Why not consider how far off the model is `abs(actual-expected)`? I wonder if there is an inflection point with that metric.
https://arxiv.org/abs/2206.07682