chapter  5
34 Pages

Replication

What’s likely to happen if you repeat an experiment? Suppose you obtain Hot Earth Awareness Test (HEAT) scores for a group of students, and calculate M = 54.3, [44.2, 64.4] and p = .072 to test the null hypothesis H0: μ0 = 45. Does the CI give useful information about the likely result of a repeat of the experiment? Does the p value? These are the two main questions I discuss in this chapter. Those results actually refer to Experiment 1, near the bottom of Figures 5.1 and 5.8, so you could, if you like, scan either of those figures for a peek ahead at what repeats of that initial experiment might give. This chapter is about what happens if you replicate an experiment over

and over. Replication is fundamental to good scientific practice, so it’s reasonable to ask what the statistical analysis of an initial experiment can tell us about what’s likely to happen on replication. I’m most interested, of course, in how CIs compare with p values in the information they give about replication. Here’s the plan:

• Replication in science. • CIs and replication. What’s the chance that the mean of a replica-

tion experiment will fall inside the CI of the initial experiment? • Dance of the capture percentages. • The sixth way to interpret a CI-as a prediction interval. • p values and replication. What information does a p value give

about what’s likely to happen next time? • Dance of the p values. • Intuitions about randomness, and why they matter.