One important caveat to all these methods is that the central limit theorem must hold for the sample means and this is an empirical condition, not something you can know statistically.
Another important caveat: many things we want to measure are not well-distributed to allow the CLT to hold. If it doesnt, the bulk of statistical methods don't work and the results are bunk.
Many quantities follow power-law distributions which would require trillions+ data points for the CLT to do its magic, ie., for the sample means of set A to be statistically-significantly different from set B would require 10^BIG if the property measured in A/B is powerlaw distributed.
Now, even worse: many areas of "science" study phenomena which is almost certainly power-law distributed, and use these methods to do so.
Another important caveat: many things we want to measure are not well-distributed to allow the CLT to hold. If it doesnt, the bulk of statistical methods don't work and the results are bunk.
Many quantities follow power-law distributions which would require trillions+ data points for the CLT to do its magic, ie., for the sample means of set A to be statistically-significantly different from set B would require 10^BIG if the property measured in A/B is powerlaw distributed.
Now, even worse: many areas of "science" study phenomena which is almost certainly power-law distributed, and use these methods to do so.