> practitioners with strong theoretical backgrounds tend to dismiss this kind of paper as "merely" engineering
This paper implements an architecture that will be out of reach for me for about 5 years. So, I ask myself "why will this paper matter in 5 years?"
There are two reasons I can imagine:
1. It shows that there is no phase change in the size-performance trend already documented over many orders of magnitude.
2. It was used as input data for pruning or distillation algorithms rooted in a better understanding of why language models work.
If the NLP community remains laser focused on hill-climbing compute-agnostic benchmarks, I don't think there will be enough people working on 2.
If 1 is the only reason it matters, I struggle to see how it is worth the cost in high-end research talent. It feels like standard low-risk corporate iteration.
This paper implements an architecture that will be out of reach for me for about 5 years. So, I ask myself "why will this paper matter in 5 years?"
There are two reasons I can imagine:
1. It shows that there is no phase change in the size-performance trend already documented over many orders of magnitude.
2. It was used as input data for pruning or distillation algorithms rooted in a better understanding of why language models work.
If the NLP community remains laser focused on hill-climbing compute-agnostic benchmarks, I don't think there will be enough people working on 2.
If 1 is the only reason it matters, I struggle to see how it is worth the cost in high-end research talent. It feels like standard low-risk corporate iteration.