I think the best way to view this paper is as a sort of meta-analysis of a wider literature around null hypothesis testing and p-values. That literature goes back at least to the 1970s with the work of people like Paul Meehl and Gene Glass. But you can push it further back, like the 1957 Lindley Paradox that Ioannidis cites.
Part of the reason this paper was impactful is that it was short and punchy, took aim at all of medicine rather than a smaller subfield, and didn't require as much mathematical understanding as other papers.
I knew some of the big names that were hit by the replication crisis. And before that I spent some time trying to talk to psychology researchers at a top school about the problems with statistical testing. But they had limited knowledge of stats and didn't want to go out on a limb when everyone else in the field seemed okay with the status quo. A paper like this can be read by everyone and makes a forceful argument.
> It then try to generalize to "research" and avoid this very narrow support to the claim
This is a good point. The methods in medicine and the social sciences are especially weak and prone to these sorts of criticisms. In the physical sciences, often you can run enough iterations of the experiment to overwhelm any prior.
> You have limitations in tech and tools you use. You might miss something, didn't have access to more information that could change the outcome. That is why research is a process.
I totally agree. Science is basically a control system, or a root finding algorithm, or gradient descent. At any time t there is a gap between the best known science and the truth. But the point is that science converges to the truth over time, whereas no other alternative does.
Part of the reason this paper was impactful is that it was short and punchy, took aim at all of medicine rather than a smaller subfield, and didn't require as much mathematical understanding as other papers.
I knew some of the big names that were hit by the replication crisis. And before that I spent some time trying to talk to psychology researchers at a top school about the problems with statistical testing. But they had limited knowledge of stats and didn't want to go out on a limb when everyone else in the field seemed okay with the status quo. A paper like this can be read by everyone and makes a forceful argument.
> It then try to generalize to "research" and avoid this very narrow support to the claim
This is a good point. The methods in medicine and the social sciences are especially weak and prone to these sorts of criticisms. In the physical sciences, often you can run enough iterations of the experiment to overwhelm any prior.
> You have limitations in tech and tools you use. You might miss something, didn't have access to more information that could change the outcome. That is why research is a process.
I totally agree. Science is basically a control system, or a root finding algorithm, or gradient descent. At any time t there is a gap between the best known science and the truth. But the point is that science converges to the truth over time, whereas no other alternative does.