Comment by mpaepper

I work in machine learning for digital Pathology and I think the big problem here is the divergence between publishing papers and real life helpful models. What we see is that in the literature you often see a model trained on data from a single lab and gets crazy good results. However, apply it to a different lab (not in the paper of course) and it sucks. So what we do in practice for our models is to train on many different labs at the same time and have a hold out test set with labs not covered during training. That way you get a robust model which works well in practice (but doesn't have the 99.9% metrics reported in papers). The second thing is looking at what task to let the machine do. We typically go for tasks which are boring / repetitive for the doctor, but visualize the result, so the doctor double checks it before making the diagnosis. Still saves a lot of time.