ABSTRACT
This chapter focuses on rater-rubric interactions, which continue to be of interest because, despite rater-training efforts, variance in rater behavior and scores persists, which may lead to reliability problems. It investigates primacy effects in relation to rater-rubric interactions. The chapter examines whether primacy effects impacted rater behavior, such as mental-rubric formation, attention to criteria, and rater scoring, when raters used an analytic rubric. It looks at how category ordering on a rubric could affect novice raters’ construction of their mental rubrics. The primacy effect seems particularly relevant for helping to explain how raters pay attention to rubric criteria. Primacy potentially impacts inter-rater reliability in using analytic rubrics. The chapter examines one key aspect of the rating process, mental-rubric formation, in order to develop a preliminary understanding of how ordering effects may influence rater beliefs. Analytic rubrics lay out detailed criteria descriptors listed by category and score band.
