ABSTRACT
This chapter describes limitations of the usability paradigm and experimental paradigm for system evaluation, usability testing, and controlled experiments, that make them poorly suited for informing and propelling the design of joint cognitive systems intended to enhance macrocognitive performance. Limitation of the experimental paradigm as a model for joint cognitive system evaluation is that it is often not possible to meet the statistical power requirements. The chapter also describes the elements and theoretical underpinnings for an alternative, work-centered evaluation paradigm, that more closely meets the goals and pragmatic constraints of conducting evaluations of complex joint cognitive systems in naturalistic environments. The work-centered approach relies on across-study replication by performing multiple evaluations or work-centered 'looks' at different points throughout the iterative design process. The focus of a work-centered evaluation is to be diagnostic in assessing whether the aid is supporting the specific macrocognitive aspects of work predicted by the model of support.
