There should be Nutrition Facts but for scientific trials. Independent agency just publishing quality assessments of the trial.
This should be an async non blocking evaluation. The statisticians who do it should be anonymous by default. There should be an appeals process for a scientist to explain why an unconventional new method is actually robust.
There should not be a single number published by this process, but rather a list of stats that speak to the overall quality of the trial on many dimensions (power, sources of bias, etc).
Only information that would not be the same on 99% of trials should be written on this label (no sec style everything is a risk word vomit disclosures).
There should not be a pre-emptive application for a label - it can only be gotten after paper submission to reduce gaming.
There should be an independent advisory org that scientists can literally call to ask for advice on structuring the trials. These calls must not be disclosed. Much like farmers can call the government to ask for help on xyz crop problem.
And these labels should never be used as the primary source of punishment. Any and all sanctions/penalties/dismissals must go through a new review process done by a different group.
Any scientist who gets a label in a particular year should be given a vote to review the review agency on several dimensions. These aggregate reviews should be published broadly but not trigger any automatic consequences.
Clear, accessible information is the basis for any self regulating human system. We need more of it in this field.
This should be an async non blocking evaluation. The statisticians who do it should be anonymous by default. There should be an appeals process for a scientist to explain why an unconventional new method is actually robust.
There should not be a single number published by this process, but rather a list of stats that speak to the overall quality of the trial on many dimensions (power, sources of bias, etc).
Only information that would not be the same on 99% of trials should be written on this label (no sec style everything is a risk word vomit disclosures).
There should not be a pre-emptive application for a label - it can only be gotten after paper submission to reduce gaming.
There should be an independent advisory org that scientists can literally call to ask for advice on structuring the trials. These calls must not be disclosed. Much like farmers can call the government to ask for help on xyz crop problem.
And these labels should never be used as the primary source of punishment. Any and all sanctions/penalties/dismissals must go through a new review process done by a different group.
Any scientist who gets a label in a particular year should be given a vote to review the review agency on several dimensions. These aggregate reviews should be published broadly but not trigger any automatic consequences.
Clear, accessible information is the basis for any self regulating human system. We need more of it in this field.