ABSTRACT
As indicated in previous topics, statistical significance deals with the question of whether a difference is reliable in light of random errors. Assume, for instance, that a researcher assigned students at random to two groups: one that was taught a mathematics lesson with new instructional software (the experimental group) and one that was taught using a traditional lecture/textbook approach (the control group). Furthermore, assume that use of the instructional software in truth is very slightly superior to the lecture/textbook approach (i.e., it really produces a superior outcome in math achievement, but the superiority is quite small). With a very large sample, randomization should yield experimental and control groups that are very similar at the beginning of the experiment (i.e., on the pretest). Because larger samples have less sampling error than smaller ones, a significance test such as a t test for the difference between the two posttest means may be able to detect the reliability of the small difference and allow the researcher to declare it to be statistically significant at some probability level, such as p < .05. This illustrates an important principle: Even a small difference can be a statistically significant difference.1 This is true because statistical significance determines only the likelihood that a difference is due to random errors, not whether it is a large difference.
