happy3483t583
Joined 3 years ago
Karma
12
Submissions
5
Member since
Oct 22
Recent submissions
Comment · 3 years ago
Comment · 3 years ago
Gateway drug: Everyone's favorite old-school t-test, brought into the modern world: https://www.sumsar.net/best_online/
Here's the story of the tool: https://jkkweb.sitehost.iu.edu/articles/Kruschke2013JEPG.pdf
(Kruschke's books and papers are a terrific resource for learning real-world applied stats)
Comment · 3 years ago
> is (thus) very sensitive to the probability of the prior
Standard practice is to test sensitivity of the posterior against various priors. They're often non-informative, e.g., `effectSize ~ N(0, wide)`, and the results are quite robust.
Here's a typical playbook: http://www.stat.columbia.edu/~gelman/research/published/4358...
Comment · 3 years ago
It's both.
I've consulted in many situations where people make horrible business decisions based on "statistically significant." People mistake p-values for p(hypothesis|data), mistake p-values for effect size ("it's highly significant!"); moreover, people using NHST don't understand multiplicity or "topping-off" problems.
Of course one could argue that they just need to review Intro Stats, but that skips over the impenetrable conceptual nature of p-values and NHST. Given that regular people must use applied stats and given that NHST is fundamentally arcane and confusing, then the stage is set for endless drama.
Now if we were sampling manufactured products from batches and looking for a quantitative upper-bound on the failure rates, run over many batches, then we have a winner!
Comment · 3 years ago
It's good but it's old-school, focusing on traditional null hypothesis significance testing and related ideas. These ideas have been under attack for decades, most recently as being one cause of the scientific replicability problem.
If you're new to applied stats, be aware that there's a whole other world out there, beautifully taught to beginners by https://xcelab.net/rm/statistical-rethinking/
Part of the ongoing stats revolution is abandoning the binary significance test, because it's often misaligned with the fundamental business question at hand. If, instead, you have a probability distribution over hypotheses, you can make optimal business decisions based on standard utility theory, like this https://philosophy.hku.hk/think/stat/naode1.php.
> It's conceptually harder than NHST
From a different angle: Experimenters fundamentally want to know p(hypothesis | data): Does adding 20g sugar to 1kg cement prevent 98% of the units from crushing under 1000kg load? Below are two answers. Which would people rather tell their boss?
1) Likely yes, given our experimental data. The totality of evidence shows that there's a 70% probability that failure rates are below the limit, but we're not yet ready to commit changes to our manufacturing. We can run more experiments to nail this down and make a concrete decision (heh).
2) Sorry boss, I have no clue. The p-value was 0.15, so we can't say anything. Unfortunately old-school stats can't easily add new data to an existing experiment, so we have to run new experiments, and then combine everything with a fancy new meta-analysis. Or we could call the first experiment a sneak peek but then we have to do multiplicity corrections. But even after we do all that, I can't really tell you the probability that sugar strengthens cement. NHST doesn't work that way.
From the stakeholder's perspective, #1 is conceptually lots easier.