Interesting. Glad to see this done, though, wonder how often large models run into these numerical bugs? They note in the paper that it's uncommon, but, I wonder if it'd affect anything still (though, I doubt it. If it was run into in practice, they would have probably shown an improvement on a larger problem than the small functions they were learning on):
> Though the extreme margins and near-zero student–teacher gaps probed by our experiments are uncommon in standard large-scale training, our fixed functions are easy drop-in replacements that can be implemented without meaningful slowdown, and may prove useful whenever models make high-confidence predictions or distill from close teachers.
> Though the extreme margins and near-zero student–teacher gaps probed by our experiments are uncommon in standard large-scale training, our fixed functions are easy drop-in replacements that can be implemented without meaningful slowdown, and may prove useful whenever models make high-confidence predictions or distill from close teachers.