ABSTRACT

This chapter is organized into seven sections that describe the behind-the-scenes, practical activities that underlie all successful standard setting activities. Although the primary focus is on standard setting for educational achievement purposes, examples from both educational and professional assessment are included. The seven topics covered include:

Identifying and nurturing stakeholders. To arrive at acceptable cut scores, those in charge of standard setting need to identify the key decision makers, determine their information needs, design a standard setting activity that addresses those needs, and gather and present information in a manner that satisfies those needs. Throughout the chapter, the reader will find suggestions for accomplishing these goals that are based on successful standard setting activities conducted in a variety of settings over many years.

Identifying and recruiting panelists. All elements of standard setting are fixed except the panelists. It is therefore critical that panelists are chosen wisely. A section on this topic describes procedures for maximizing the representativeness of the standard setting panel and thereby the validity of the recommended cut score(s).

Scheduling. Standard setting involves knowing when key decisions have to be made and how to make sure decision makers have the information they need when they need it. A section on scheduling provides suggestions for involving facilitators, panelists, decision makers, and other stakeholders in the scheduling of standard setting activities.

Preparing materials. The person in charge of the standard setting activity prepares forms for panelists to complete, training materials, computer programs, feedback data sheets, impact tables and graphs, and other materials. The section on this topic describes a variety of materials, their functions, and methods of preparation.

Conducting standard setting. Regardless of specific method employed, there is the task of leading a group of panelists to a valid conclusion. The section on this activity offers suggestions for keeping the panel on task and deals with facilities, equipment, work flow, security, and related issues.

Reporting results. Through careful documentation of the needs of decision makers, the standard setting facilitator or organizer is able to prepare a concise, meaningful report that will have maximum impact. The section on reporting results describes the steps to take to achieve this goal and offers a generic outline and rationale for a final report.

Follow-up activities. Even after the standard setting activity is over and the report is written, there are still activities to complete before cut scores are accepted and implemented. These may include review of results with stakeholders, presentations to governing boards, and other events. A section describes a process for identifying the necessary activities and carrying them out successfully.