ABSTRACT
This chapter introduces a variety of means of assessing reliability of measurement. It discusses the most influential theory of measurement known as classical test theory and shows how reliability of measurement is defined under the tenets of classical test theory. An estimate of reliability derived from a correlation between observed measurements over time is known as a test-retest reliability estimate, for obvious reasons. The reliability of other researchers’ measurements can be used to justify the selection of the method of measurement, but it is not relevant to evaluating the quality of data. Low reliability exists when units are routinely miscategorized, meaning many categorization errors. High reliability exists when the units are by and large correctly classified. In short, low agreement brings into question the reliability of the data and ultimately the generalizability of any research findings. The key to justifying exchangeability is the size of the agreement between two coders who are asked to code a subset of the units.
