ABSTRACT
Effect size and statistical significance are imperfectly related to one another. The greater the variability of measures, the less likely they are to be significantly different. To get an accurate estimate of the chance that a hypothesis is true, in the sense of replicable, the sample space must be known exactly. The hypothesis to be tested, the null hypothesis, is that some groups are the same—that any measured difference between them could have arisen by chance. Hence, there is no alternative to testing the null hypothesis with the aid of what is in effect another hypothesis: a model of the sample space. Experiments with groups of subjects require statistical tests to evaluate their results. The statistics are the same as those used to compare groups of any kind, including those derived without the aid of experiment. Even if the probabilities yielded by standard statistical methods are accurate, there is still a serious problem with the null hypothesis statistical test method.
