ABSTRACT
This chapter presents and discusses the core processes required in developing and administering language tests and makes the case for adopting a quality management (QM) approach to improve testing and to ensure that appropriate professional standards are met. In this respect, language test developers need to adopt the kind of managerial practices which enable successful organisations to implement error-free processes. In making this case, it is important to define what quality means in the context of language
assessment and to describe how an approach based on QM can help to achieve the required quality goals and at the same time to enhance a validity argument (see Kane, this volume). By adopting that QM approach, it is possible to ensure that processes are continually improved and standards raised; this is in keeping with the concept of validation as a long-term, ongoing activity (Kane, 2006). A key point is the necessary convergence between the twin concepts of quality and validity.
I argue that QM provides an appropriate basis for guaranteeing the consistency of the processes which underpin a validity argument and provides the tools and techniques for linking theoretical concerns with the practical realities of setting up and administering assessment systems. The principles of QM and the link with validation can be widely applied in many different
test development contexts with the aim of improving the quality of assessment systems. A number of explicit stages need to be followed from production of the test materials to the issuing of the test results. This applies whether the test development team is a group of teachers producing language tests for students within their own classes, or whether they are specialised staff within a testing agency responsible for developing and administering high stakes, international language examinations in many countries. To illustrate the applicability of the approach, these two contexts – classroom assessment and
large-scale assessment – will be referred to throughout this chapter. In both cases the validity of the assessment needs to be ensured and the integrity of the results guaranteed for the intended purposes, even if these may be very different. To achieve the required validity profile for the intended purpose and use, the quality of the development and administration procedures have to be checked and monitored at each stage of the assessment cycle (see Figure 27.1). What
differs between the school context and the large-scale assessment operation is the complexity of the organisational structure within which the activities take place and, most obviously perhaps, the resources available for achieving the intended assessment goals, i.e., the human, material and time-related factors which are available to meet the intended objectives. Other factors may relate to the intended uses of the test results and the importance attached to decisions made about the test takers. In other words whether the test is considered to be high or low stakes by the users and whether this may offer opportunities to increase the resources available (e.g. in order to achieve adequate reliability of test scores to support high stakes decisions about test takers). Practical considerations are always important. The test developer must attempt to match the
ambitions of the assessment sponsors to the ‘real world’ considerations and constraints which exist in their context. This inevitably requires judgments to be made about deploying the available resources in the most effective way at all stages of the cycle. Careful planning is required at the outset, and appropriate management systems need to be set up to allocate and manage resources throughout. Leadership and communication skills are important and those involved in a test development project need to employ techniques for managing successful projects and to be aware of critical issues related to the management of change within organisations. Bachman and Palmer (1996: 35-37) discuss practicality as an important ‘test quality’ and Bach-
man and Palmer (2010: 411-27) dedicate a chapter to the identification, allocation and management of resources. These points are taken up below when considering aspects of organisational management.
