rwilson4

This is a complex topic, but it's a bit simpler when outcomes are bounded, such as a binary outcome that either occurs or does not occur. In that case, the impact of any one factor is bounded.

In the scenario you're describing, this other factor drowns out any influence the treatment has on the outcome. You'll struggle to get a statistically significant result (low power) and the confidence interval on the treatment effect will include 0. This too can be a valuable finding: sometimes the answer is that the treatment is not particularly effective.

The author is using the technical definitions of confounders and covariates without sufficient explanation, and the technical definitions do not match the normal English definitions.

In English, a confounder is any factor that distorts an observation. (My dictionary defines it as throwing into confusion or disarray.)

In causal inference, a confounder is a factor that is correlated with both treatment and outcome. If the treatment is randomly assigned, by construction it is independent of all other factors. This, there can be no confounders.

Your example is about observed occurrences of imbalance, but the technical definition is about probabilities. Observed imbalances can still skew inference, but that causes high variance (or low precision). It doesn't cause bias (or affect accuracy).

Adjusting for observed imbalances can reduce variance, but in some circumstances can actually cause bias.

In the hard sciences, it's often possible to isolate the phenomenon of interest away from any other influencing factors, e.g. in a laboratory. But many phenomena, like social interactions, or even agriculture, are difficult to isolate in this way. Randomization provides another way of "zooming in" on the treatment of interest.

In the example you gave, a test is going to have very low power because of the important factor with huge variance. If that factor is observed, you can create pairs of units with that factor identical within the pair, then randomly assign treatment to one unit in each pair.

As someone who works with causal inference most days, I expected to find much fault with this article. I was pleasantly surprised to find how rigorous the article is, despite some other comments here. For more information on the role of randomization in causal inference (experimental and observational), I recommend the books by Paul Rosenbaum, especially, "Observation and Experiment".

Thanks, Bastien, for org-journal. I've used it almost every day for the last 4 years. Perhaps you'll find a new use for Emacs someday.

The textbook, "Bayesian Data Analysis" by Gelman et al, has a good discussion on this in Chapter 8. Here are some relevant bits:

"A naive student of Bayesian inference might claim that because all inference is conditional on the observed data, it makes no difference how those data were collected. This misplaced appeal to the likelihood principle would assert that given (1) a fixed model (including the prior distribution) for the underlying data and (2) fixed observed values of the data, Bayesian inference is determined regardless of the design for the collection of the data. Under this view there would be no formal role for randomization in either sample surveys or experiments."

"The notion that the method of data collection is irrelevant to Bayesian analysis can be dispelled by the simplest of examples. Suppose for instance that we, the authors, give you, the reader, a collection of the outcomes of ten rolls of a die and all are 6's. Certainly your attitude toward the nature of the die after analyzing these data would be different if we told you (i) these were the only rolls we performed, versus (ii) we rolled the die 60 times but decided to report only the 6's, versus (iii) we decided in advance that we were going to report honestly that ten 6's appeared but would conceal how many rolls it took, and we had to wait 500 rolls to attain that result."

Out of curiosity, how do you self-blind? Do you have a friend make your coffee for you, with/without the additive?

That's me! My primary language is python, but after that I write more elisp than anything else.

I played around with different tap&hold durations (how long you have to hold before it registers as a hold rather than a tap), but I just couldn't find a duration that matched my rhythm. I kept getting holds when I wanted taps (for short durations) or vice versa for longer durations. My current config uses tap&hold on two keys (home and end, which I don't use often anyway). Holding leads to "hyper" and "meh" which are two extra modifier keys (like control or alt/option/meta). I'm an emacs user so extra modifiers means I can basically store as many macros as I want!

I have a Kinesis Advantage 2 and the thumb clusters are my favorite part. I think I probably spend too much time optimizing the layout of those keys though. Like, Escape is really awkwardly placed in the default layout so I moved it to the right thumb cluster, but I was actually just thinking about moving it to the left cluster. Talk about micro-optimizations!

There are some great technical solutions offered, but the real one is to publish your data and your code, and be as transparent as you can about what you did.

Agreed. But as Cox and Hinkley discuss, the likelihood principle is sometimes at odds with the repeated sampling principle, so in any particular application, you need to identify if there is a conflict, and if so, which principle is more important. In my domain (simple A/B tests), you can claw the repeated sampling principle from my cold, dead hands.

There is less of a conflict than many would have you believe. In many situations, both approaches yield the same answer. There are some edge cases. For example, in A/B testing, is early peeking bad? From a frequentist perspective the answer is "yes, either use a sequential method, or don't early peek at all". From a Bayesian perspective the answer is "early peeking is fine".

It boils down to what properties you want your analysis to have. Cox and Hinkley's "Theoretical Statistics" has a great discussion (section 2.4). Basically, you might want your analysis to have a certain kind of internal consistency. But you might also want your analysis to be replicable either by yourself or by another researcher. Those both seem like pretty important things! But there are edge cases (like the early peeking example) where you can't have it both ways. So you have to pick which one you want, and use the corresponding methods.