blackbear_

As we all know, oeer review is not a very reliable signal about the veridicity of the claims, it is merely an approximate quality gate indicating that a study was done following appropriate standards. There needs to be more awareness of this. Nonetheless, in some fields peer review have been public for a long time, and more and more journals are making peer reviews publicly visible. This is a good trend that we should all encourage.

I also think that it sounds like a good idea to hold scientists accountable for the quality of their research, but moving away from fraud (which should unquestionably be punished) towards merely "low quality" is a very slippery slope. Who determines when the quality is good enough and under which criteria? Citations and follow-up research already are an approximate signal, why is that not enough? Proving a study false does not automatically make it low quality. How much money and resources would these additional measures cost? Would the savings from fewer "bad quality studies" offset these costs? How many fewer "good quality studies" would be prevented by redirecting these resources.

Totally agree with this framing: trust in public institutions (scientific, governmental, etc.) is a common good that needs to be promoted and safeguarded.

Identifying and punishing fraud is a part of it, whistleblower protection another, modernizing the system so that graduate students are not so dependent on individual advisors yet another idea. More and more personal contact between the public and scientists could be useful too, for example via science fairs, meet & greets and so on.

Yup, that is also how it works with scientific research: Once an article is published, anybody is free to read it, criticize it, reproduce it. That's because the government stays out of it: private individuals can use things like Google Scholar and Google Search to find other people's reviews and follow-up studies to better understand the quality and nuances of the original research. Consensus is an important part of science just as it is for reviews on Google Maps. Is consensus sometimes wrong? Yes, no doubt. Is it useless? Nope.

> If people can't define it, then that seems to me like a valid criticism of the scientific method.

Please read the hamburger example once more.

This is unnecessarily cynical.

> The core problem is people conflating science with Science™.

You fell into the same trap by criticizing scientific institutions and the human spirit in an attempt to criticize the scientific method.

> What exactly is the scientific method?

What a ridiculous argument. What exactly is a hamburger? Just ask around and you will find that nobody agrees if a veggie burger should be considered a hamburger, or if you can replace the bun with lettuce and still call it a hamburger. And even if you get people to agree what a hamburger is, nobody is checking that the recipe is followed to the letter. Therefore, real hamburgers do not exist, and nobody can claim in good faith that they are making a hamburger. In fact, why don't we sue McDonalds for their fraud hamburgers?

> Science isn't self correcting.

The very fact that fraud is identified is clear evidence that is is not true. You are going to need to sharpen your rethorical devices if you want to argue against hundreds of years of scientific progress, including for example everything that enabled us to argue about this in front of our devices while barely 150 years ago people had to use candles to see each other after dinner. Mind you, not all of this was developed by industrial research labs, in fact the origins of very many modern technologies lie in academic research labs pursuing ideas that are too risky for a commercial entity to bet on. And of course there are plenty of examples of industrial fraud happening despite "managerial intervention" (lol).

Reality is more complex. Not many things, if at all, exist in this world under their platonic ideal form. That does not make them any less real. Yes, we all know that humans are not perfect, sometimes are selfish and chase money and power, not only in academia but also in business, politics, and so on. Wherever humans are involved this sometimes happens. But saying that nothing good ever happens because of this goes against everything one can see when looking out of the window. Scientific progress is real, and needs both industrial and academic labs.

This was abundantly clear when people, even on HN, were upset about the bureau of labor statistics revising their numbers tendentially downwards, probably confusing the notion of statistical bias for that of political bias.

If the figures are biased, just estimate the bias and correct for that, what is the big deal they said, as if the bias variance tradeoff was not a thing.

Having similar thoughts at times. What are you reading nowadays?

There are other players in the game: the business and the market.

Good code makes it easier for the business to move fast and stay ahead of the competition while reducing expenses for doing so.

Is it because people genuinely don't care, or because the barrier to become a power user is becoming taller and taller every passing year?

> what kind of neural-net architecture and training would allow a model to handle numbers lengths it hasn't been trained on

A recurrent neural network implementing binary addition with carry could do this, and one can derive the correct weights with pen and paper without too much effort.

Whether gradient descent will find them too is another matter entirely

> I measure, test, and validate outputs exhaustively.

How do you do this? Do you follow traditional testing practices or do you have novel strategies like agents with separate responsibilities?