Regarding replicability, I disagree this is a problem at all. Writing shit code is not going to prevent someone highly capable from replicating your results. If anything, I empathize with researchers writing sloppy code. It’s a creative field and they already have to do enough editing and documentation. Omitting code or fabricating/manipulating evaluation results is what prevents replication.
Frankly, unlike the author, I think there’s too many people in the field. They produce a handful of papers worth reading every year along with thousands upon thousands of models that may or may not slightly improve performance on a specific task and then have no general value beyond that. And I don’t believe this will change much- ml is likely the most monetizable PhD path by a safe margin, so there is too much profit incentive to churn out crap at any cost.
Frankly, unlike the author, I think there’s too many people in the field. They produce a handful of papers worth reading every year along with thousands upon thousands of models that may or may not slightly improve performance on a specific task and then have no general value beyond that. And I don’t believe this will change much- ml is likely the most monetizable PhD path by a safe margin, so there is too much profit incentive to churn out crap at any cost.