Module 10. Validity Generalization and Psychometric Meta-Analysis
Schmidt and Hunter (2001) discussed the myth of the “perfect study.” That is, if we could somehow get a large enough sample with perfectly reliable and valid measures, then we could definitively answer the key and nagging questions plaguing the social and behavioral sciences. Although some large-scale studies have been conducted with thousands of participants, most individual empirical studies, particularly in psychology, tend to average in the hundreds (or fewer) of participants, not thousands. As a result, sampling error is a major source of error in estimating population relationships and parameters within any given empirical investigation. In addition, a variety of factors (i.e., methodological artifacts), such as unreliable measures, restriction of range, and artificial dichotomization of continuous variables, are an undeniable part of any individual empirical investigation. In the end, such artifacts cloud our observed relationships and ultimately our abi - lity to estimate population relationships based on sample data. Therefore, it is simply unrealistic to believe that any single study is going to be able to definitively explain the complex relationships found among key variables in the social and behavioral sciences. So, what is a budding social and behavioral scientist to do?