ABSTRACT
Hypothesis testing has an odd logic. We collect a sample, calculate a statistic, and produce a probability of obtaining it or a more extreme value. From a naïve perspective, the probability of that sample is 1. It happened. This is not unlike picking an item from a container with items in unknown proportions and then asking about the probability of picking the item you picked. Of course, to sophisticates of the subject this is silly. They know that a probability statement about a sample’s statistic is not really about that sample. It is about the process of collecting sample statistics from a population of values having an assumed distribution.1 Velleman (1997) addresses a related issue nicely when he asks and answers his own question, “Where is the randomness?” in regard to a confidence interval. He says,
Velleman’s explanation clarifies that “90% confidence” is not a claim about a specific interval, but rather is a claim about the method by which such intervals are produced. Similar conceptions are at the foundation of hypothesis testing, except that hypothesis testing draws on the logic of indirect argument. We assume that all possible values of the test statistic are distributed somehow, centered at the population parameter, and gauge whether the value we obtained is sufficiently unusual relative to those assumptions that it puts them in doubt. If so, then we conclude that our assumptions are faulty.
