> p=0.05 means that one result in 20 is going to be the result of chance.
You made the same mistake most people make here: you turned the arrow of the implication. It is not "successful experiment implies chance (probability 5%)" but "chance implies successful experiment (probability 5%)".
What does that mean in practice? Imagine a hypothetical scientist that is fundamentally confused about something important, so all hypotheses they generate are false. Yet, using p=0.05, 5% of those hypotheses will be "confirmed experimentally". In that case, it is not 5% of the "experimentally confirmed" hypotheses that are wrong -- it is full 100%. Even without any cherry-picking.
The problem is not that p=0.05 is too high. The problem is, it doesn't actually mean what most people believe it means.
You made the same mistake most people make here: you turned the arrow of the implication. It is not "successful experiment implies chance (probability 5%)" but "chance implies successful experiment (probability 5%)".
What does that mean in practice? Imagine a hypothetical scientist that is fundamentally confused about something important, so all hypotheses they generate are false. Yet, using p=0.05, 5% of those hypotheses will be "confirmed experimentally". In that case, it is not 5% of the "experimentally confirmed" hypotheses that are wrong -- it is full 100%. Even without any cherry-picking.
The problem is not that p=0.05 is too high. The problem is, it doesn't actually mean what most people believe it means.