> History teaches us that end to end learned system beats human designed mechanisms
Depends how far back you go. History teaches us that everything is a trade off between model size, inference time, training time, and training data size, Once you're at the pareto frontier. And that cheap approximations can allow you to trade for more expensive computation elsewhere.
That lesson has been obscured for the last decade because (1) "the bitter lesson" of scaling, and, (2), we're blowing past benchmarks too quickly.
I do agree that learned models are better if they're free (compare the distribution of filter banks learned by a neutral acoustic model to those approximated by mel frequency cepstral coefficients), but once you start hitting scaling limits, cheap heuristics start creeping back in.
BPE was a huge advancement over fixed vocab, e.g.
Depends how far back you go. History teaches us that everything is a trade off between model size, inference time, training time, and training data size, Once you're at the pareto frontier. And that cheap approximations can allow you to trade for more expensive computation elsewhere.
That lesson has been obscured for the last decade because (1) "the bitter lesson" of scaling, and, (2), we're blowing past benchmarks too quickly.
I do agree that learned models are better if they're free (compare the distribution of filter banks learned by a neutral acoustic model to those approximated by mel frequency cepstral coefficients), but once you start hitting scaling limits, cheap heuristics start creeping back in.
BPE was a huge advancement over fixed vocab, e.g.