ABSTRACT
This chapter discusses practical issues for rater-mediated assessments, with a focus on issues related to designing rater-mediated assessment systems from the perspective of invariant measurement. It explores the constructing measures framework and its adapted version to the context of rater-mediated assessments. The distinction between the first building block and building block included in the original presentation of the constructing measures framework is related to the definition of the construct as rater-mediated. Using an adaptation of M. Wilson's constructing measures framework, the chapter provides an overview of key issues related to the design of rater-mediated assessment systems within the framework of rater-invariant measurement. The finding of closer correspondence when strong links between facets in an assessment system are maintained highlights the importance of establishing adequate links between raters in order to obtain rater-invariant performance estimates. Recognizing issues related to rater exchangeability, many performance assessment systems include training programs with qualification requirements prior to operational scoring.
