ABSTRACT
In this chapter we will begin by defining prototyping and differentiating it from other terms used to indicate tryouts of test materials before they are used operationally. Next, we will consider what the characteristics of a prototyping population should be, and, finally, we will discuss what kinds of information prototyping can provide. We will use extended examples to illustrate the issues addressed in the study. Although prototyping is carried out by many language testing programs, the examples in this chapter are from tests designed at Educational Testing Service (ETS) only. The authors, both on the test development staff of ETS, have worked on numerous prototyping projects and will draw on that experience to provide the examples in this chapter. In a language testing context, prototyping refers to trying out new test materials for their appro-
priateness for use prior to field testing and making final decisions about the operational test content. Prototyping may provide information about the design of a new test or about revisions to an ongoing test. Before prototyping begins, a test construct must exist and the abilities to be measured that are part of the construct must be carefully considered and documented. By prototyping items we can investigate the extent to which our theories and expectations about what the items measure are accurate (e.g., by comparing performance of examinees to their performance on other measures of the construct) and the extent to which construct-irrelevant variance occurs because of item design (e.g., by interviewing examinees about their prototype experience). Numerous factors contribute to decisions as to the amount and type of prototyping that is
needed. If the item types are novel and little is understood about how examinees might perform on them, more extensive prototyping is likely to be necessary, and the issues to be addressed may be different than they would be for the introduction of a well-known item type into a revised test. For a revised test, the reasons for the revision (e.g., to improve reliability and/or discrimination, or to bring the test more in line with current understanding of the construct) will likely determine the types of prototyping questions to be addressed. In addition to “prototyping,” a number of other terms are used to describe various ways of
trying out new items and tasks before they can be considered for use operationally. Some of these terms are used interchangeably with each other and with the term prototyping, and some indicate real differences in the purpose or stage of pre-operational administration.
