Comment by cgn

I am an author on the paper, and just saw this thread. Thanks for all of your comments. Note this work focuses on errors in test sets, not training labels. Test sets are foundation of benchmarking for machine learning. We study test sets (versus training sets), because training sets are already studied in-depth in confident learning (https://arxiv.org/pdf/1911.00068.pdf) and because before this paper, no one had studied (at scale, for lots of datasets) if these test sets had errors in the first place (beyond just looking at the well-known erroneous ImageNet dataset, for example) and do these test sets affect benchmarks. The answer is, yes, if the noise prevalence is fairly high (above 10%). The takeaway, is you must benchmark models on clean test sets, regardless of noise in your training set, if you want to know how your model will perform in the real world.