I think we're actually in violent agreement here, but I just wasn't precise enough. Let me try again:
p=0.05 means that one POSITIVE result in 20 is going to be the result of chance and not causality
In other words: if I have some kind of intervention or treatment, and that intervention or treatment produces some result in a test group relative to a control group with p=0.05, then the odds of getting that result simply by chance and not because the treatment or intervention actually had an effect are 5%.
The practical effect of this is that there are two different ways of getting a p=0.05 result:
1. Find a treatment or intervention that actually works or
2. Test ~20 different (useless) interventions. Or test one useless intervention ~20 times.
A single p=0.05 result in isolation is useless because there is no way to know which of the two methods produced it.
This is why replication is so important. The odds of getting a p=0.05 result by chance is 5%. But the odds of getting TWO of them in sequential trials is 0.25%, and the odds of a positive result being the result of pure chance decrease exponentially with each subsequent replication.
The practical effect of this is that there are two different ways of getting a p=0.05 result:
1. Find a treatment or intervention that actually works or
2. Test ~20 different (useless) interventions. Or test one useless intervention ~20 times.
A single p=0.05 result in isolation is useless because there is no way to know which of the two methods produced it.
This is why replication is so important. The odds of getting a p=0.05 result by chance is 5%. But the odds of getting TWO of them in sequential trials is 0.25%, and the odds of a positive result being the result of pure chance decrease exponentially with each subsequent replication.