ABSTRACT
This chapter differs from the other two in Part VI of the book in two respects: first, it focuses on assessing a particular area of language knowledge, vocabulary, and second, it deals with the somewhat ill-defined topic of pilot testing. By contrast, the other chapters range more widely over types of items and tasks, and they deal with steps in the test development process that are better defined in language assessment, at least in these chapters themselves and in Fulcher and Davidson (2007: 76-90). Let us take up the latter point first. Various authors (see, e.g., Bachman and Palmer, 1996:
248; Moussavi, 1999: 284) have pointed out that there is a confusing array of terms which are partly if not fully synonymous – ‘pilot’, ‘pre-test’, ‘trial’ and ‘tryout’ – referring to a particular event or activity, along with the corresponding set of verbal forms – ‘piloting’, ‘pre-testing’, etc. – that refer to a more general process. As it happens, the two sources just cited take ‘pretest/ing’ to be the default term, as do Alderson et al. (1995: 74), although the latter authors see it as a cover term for two distinct phases, which they call ‘pilot testing’ and ‘main trials’. To add to the confusion, another popular testing book for teachers (Hughes, 2003) identifies the same two phases as Alderson et al. but refers to them both as ‘trials’. In their dictionary, Davies et al. (1999) have separate main entries for ‘pre-test’ and ‘trial(ling)’, but they acknowledge that the terms are not clearly distinguished in the literature. Thus, unfortunately, as in so many other areas of applied linguistics, we have to live with a lack of standardization in the terminology that is used. I will adopt the cover terms ‘pilot test’ and ‘pilot testing’ here, but they should be understood as referring broadly to the same kinds of activity as the other terms. What is probably more important is to be aware of how these activities are planned to improve the quality of draft test material before it is used for assessment purposes. Of that, more later. It is also necessary to define the scope of vocabulary testing. Vocabulary is obviously an
indispensable component of language use, to the extent that almost any kind of language test is in a sense measuring the test-takers’ vocabulary ability, even if this ability is not an explicit focus of the scoring or rating. I distinguish between ‘discrete’ and ‘embedded’ assessments of vocabulary knowledge and use (Read, 2000: 8-10). The embedded type involves a focus on vocabulary as part of the assessment of a broader construct, as when a reading comprehension test
includes some items that measure whether the test-takers can understand particular words or phrases that occur in the reading passages, or when the raters in a speaking test evaluate the range and accuracy of vocabulary use as one of the criteria for rating each candidate’s performance. However, normally vocabulary tests are a discrete form of assessment, in that they are intended to focus specifically on some aspect of vocabulary knowledge or use, based on a pre-selected set of words. This chapter will be limited to the discussion of discrete vocabulary testing in this sense.
