ABSTRACT
This chapter focuses on how to design evaluation scenarios that adequately probe the boundaries of effectiveness of the support system. It describes an approach to scaling up the complexity of existing scenarios based upon a working model of what facets make up complexity in supervisory control domains. The chapter discusses the multiple levels of description that need to be considered when designing scenarios, and presents a domain-independent set of cognitive and collaborative 'complicating factors' that can be used to guide the design of evaluation scenarios. These complicating factors can be embedded individually or in clusters into existing scenarios so as to exercise the software under conditions that accurately reflect the types of complexities that may arise in the field. The chapter aims to view these factors as comprising a working definition of the facets that contribute to complexity in dynamic, high-risk, real-world settings where cognition is distributed across human agents supported by sophisticated artifacts.
