I agree that peer review can be a strong filter, but it's a filter for claims and evidence that sound true. CS papers can and do hide important details in the code (details which, I argue, would get a paper rejected if they were stated in the paper).
Regardless of the strength of the filter, if the filter's inputs are just "the paper", but the claims depend on the details in another artifact (i.e. the code), how can we argue that peer review filters for the truth?
Regardless of the strength of the filter, if the filter's inputs are just "the paper", but the claims depend on the details in another artifact (i.e. the code), how can we argue that peer review filters for the truth?