Test Scores as Samples

Connections With Other Chapters

In chapter 3, generalizability theory was introduced. This theory interprets test scores as samples from a universe of observations. The corresponding conceptualization of the relation between test score and construct is given by behavior domain theory. This theory is interesting because it appears compatible with an approach to validity that makes no reference to causation. The present chapter develops such an interpretation and investigates its tenability. It is argued that behavior domain theory relies on implicit causal relationships, albeit minimal, to allow stable generalizations from test score to behavior domain. In addition, it is compatible with causal theories in three ways. First, one may take a causal structure as providing the basis for a homogeneous domain. Second, one can construct a homogeneous domain and then investigate whether a causal structure explains the homogeneity. Finally, one can take the domain score as linked to an existing attribute constrained by indirect measurement.

This chapter ¹ explores the possibility of reconciling behavior domain theory (BDT) with a causal theory of measurement (CTM). In BDT, constructs are conceptualized in terms of domains of behavior, and item responses are considered samples from this domain. One can see BDT as the conceptual counterpart of generalizability theory, as discussed in chapter 3. As in generalizability theory, the relation between behaviors in the domain and item responses in the test is a sampling relation. This makes the inference from item scores to construct scores a generalization of the population-sample variety. In CTM, constructs refer to common causes (equivalently, attributes; Rozeboom, 1966) that underlie a set of item responses, so that people respond to items differently because they have a different construct score (Borsboom, 2008; Borsboom, Mellenbergh, & Van Heerden, 2004). In this case, conclusions about constructs, on the basis of item responses, require causal inference rather than generalization.