ABSTRACT
This chapter examines one way that scientific quality is operationalized in psychology and other fields. The natural consequence of these two methods of modifying science is that they imply two separate criteria for scientific quality. The first criterion for scientific quality examines the logic of scientific work from the perspective of current theory, and how well that work addresses some facet of theory. The second criterion for scientific quality stems from Kuhn's revolutionary science. The Gatekeeper role is performed ineffectively largely because the external experts who were judging scientific quality were in such poor agreement. Assessing provocativeness would probably engender even lower reviewer agreement than is found because being provocative involves two facets: the contents of the article and the readers' response. Reviewers' judgments can be examined with respect to how well they predict reader feedback and an article's future citations.
