This is entirely unsurprising and has a very simple solution: keep adding more data. Our measurements of the accuracy of AI systems are only as good as the test data, and if the test data is too small, then the reported accuracies won't reflect the true accuracies of the model applied to wild data.
Basically, we need an accurate measure of whether the test data set is statistically representative of wild data. In healthcare, this means that the individuals that make of the test dataset must be statistically representative of the actual population (and also have enough samples).
An easy solution here is that any research that doesn't pass a population statistics test must be up-front declared to be "not representative of real word usage" or something.
Basically, we need an accurate measure of whether the test data set is statistically representative of wild data. In healthcare, this means that the individuals that make of the test dataset must be statistically representative of the actual population (and also have enough samples).
An easy solution here is that any research that doesn't pass a population statistics test must be up-front declared to be "not representative of real word usage" or something.