ABSTRACT
Limitations of statistical significance testing Concentration on statistical significance misses an important aspect of inferential statistics-statistical significance is affected by sample size. This has two consequences. Firstly, statistical probability cannot be used as a measure of the magnitude of a result; two studies may produce very different results, in terms of statistical significance, simply because they have employed different sample sizes. Therefore, if only statistical significance is reported, then results cannot be sensibly compared. Secondly, two studies conducted in the same way in every respect except sample size may lead to different conclusions. The one with the larger sample size may achieve a statistically significant result while the other one does not. Thus, the researchers in the first study will reject the Null Hypothesis of no effect while the researchers in the smaller study will reject their research hypothesis. Accordingly, the smaller the sample size, the more likely we are to commit a Type II error-rejecting the research hypothesis when in fact it is correct.
