ABSTRACT
To illustrate how to use the framework described in Chapters 23 and 24 to evaluate research validity, we now evaluate the ve studies introduced in Chapter 1. They vary in the approach used but have a number of features in common: They all were published in peer-reviewed journals, made a reasonable case for the research questions and methods based on literature and theory, and interpreted the results with appropriate caution given the strengths and weaknesses of the methods used. Thus, Q17-Q19 (other issues) probably would not produce major differences in the evaluations for this set of peer-reviewed articles. However, there were differences in the strength of various aspects of the design and methods, as discussed next. We evaluate each of the studies in order using global judgments and narratives to rate each of the eight research validity scales (Q9-Q16) described in Chapters 23 and 24.1
There is, of course, some degree of subjective judgment involved both with the eight ratings used in this chapter and when using the 2006 numerical evaluation form. Thus, different knowledgeable evaluators might produce different ratings for the eight key dimensions (Q9-Q16). However, we believe that reliability would be quite high and that the mean difference in ratings would be small.
